Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
Test::LectroTest::TestUsereContributed Perl DocTest::LectroTest::TestRunner(3)

NAME
       Test::LectroTest::TestRunner - Configurable TAP-compatible engine for
       running LectroTest property checks

VERSION
       version 0.5001

SYNOPSIS
	use Test::LectroTest::TestRunner;

	my @args = trials => 1_000, retries => 20_000;
	my $runner = Test::LectroTest::TestRunner->new(	@args );

	# test a single	property and print details upon	failure
	my $result = $runner->run( $a_single_lectrotest_property );
	print $result->details unless $result->success;

	# test a suite of properties, w/ Test::Harness::TAP output
	my $num_successful = $runner->run_suite( @properties );
	print "# All passed!" if $num_successful == @properties;

DESCRIPTION
       STOP! If	you just want to write and run simple tests, see
       Test::LectroTest.  If you really	want to	learn about the	property-
       checking	apparatus or turn its control knobs, read on.

       This module provides Test::LectroTest::TestRunner, a class of objects
       that tests properties by	running	repeated random	trials.	 Create	a
       TestRunner, configure it, and then call its "run" or "run_suite"
       methods to test properties individually or in groups.

METHODS
       The following methods are available.

   new(named-params)
	 my $runner = new Test::LectroTest::TestRunner(
	   trials      => 1_000,
	   retries     => 20_000,
	   scalefn     => sub {	$_[0] /	2 + 1 },
	   verbose     => 1,
	   regressions => "/path/to/regression_suite.txt",
	 );

       Creates a new Test::LectroTest::TestRunner and configures it with the
       given named parameters, if any.	Typically, you need only provide the
       "trials"	parameter because the other values are reasonable for almost
       all situations.	Here is	what each parameter means:

       trials
	   The number of trials	to run against each property checked.  The
	   default is 1_000.

       retries
	   The number of times to allow	a property to retry trials (via
	   "$tcon->retry") during the entire property check before aborting
	   the check.  This is used to prevent infinite	looping, should	the
	   property retry every	attempt.

       scalefn
	   A subroutine	that scales the	sizing guidance	given to input
	   generators.

	   The TestRunner starts with an initial guidance of 1 at the
	   beginning of	a property check.  For each trial (or retry) of	the
	   property, the guidance value	is incremented.	 This causes
	   successive trials to	be tried using successively more complex
	   inputs.  The	"scalefn" subroutine gets to adjust this guidance on
	   the way to the input	generators.  Typically,	you would change the
	   "scalefn" subroutine	if you wanted to change	the rate and which
	   inputs grow during the course of the	trials.

       verbose
	   If this paramter is set to true (the	default) the TestRunner	will
	   use verbose output that includes things like	label frequencies and
	   counterexamples.  Otherwise,	only one-line summaries	will be
	   output.  Unless you have a good reason to do	otherwise, leave this
	   parameter alone because verbose output is almost always what	you
	   want.

       record_failures
	   If this parameter is	set to a file's	pathname (or a FailureRecorder
	   object), the	TestRunner will	record property-check failures to the
	   file	(or recorder).	(This is an easy way to	build a	regression-
	   testing suite.)  If the file	cannot be created or written to, this
	   parameter will be ignored.  Set this	parameter to "undef" (the
	   default) to turn off	recording.

       playback_failures
	   If this parameter is	set to a file's	pathname (or a FailureRecorder
	   object), the	TestRunner will	load previously	recorded failures from
	   the file (or	recorder) and use them as additional test cases	when
	   checking properties.	 If the	file cannot be read, this option will
	   be ignored.	Set this parameter to "undef" (the default) to turn
	   off recording.

       regressions
	   If this parameter is	set to a file's	pathname (or a FailureRecorder
	   object), the	TestRunner will	load failures from and record failures
	   to the file (or recorder).  Setting this parameter is a shortcut
	   for,	and exactly equivalent to, setting record_failures and
	   <playback_failures> to the same value, which	is typically what you
	   want	when managing a	persistent suite of regression tests.

	   This	is a write-only	accessor.

       You can also set	and get	the values of the configuration	properties
       using accessors of the same name.  For example:

	 $runner->trials( 10_000 );

   run(property)
	 $results = $runner->run( $a_property );
	 print $results->summary, "\n";
	 if ($results->success)	{
	     # celebrate!
	 }

       Checks whether the given	property holds by running repeated random
       trials.	The result is a	Test::LectroTest::TestRunner::results object,
       which you can query for fined-grained information about the outcome of
       the check.

       The "run" method	takes an optional second argument which	gives the test
       number.	If it is not provided (usually the case), the next number
       available from the TestRunner's internal	counter	is used.

	 $results = $runner->run( $third_property, 3 );

       Additionally, if	the TestRunner's playback_failures parameter is
       defined,	this method will play back any relevant	failure	cases from the
       given playback file (or FailureRecorder).

       Additionally, if	the TestRunner's record_failures parameter is defined,
       this method will	record any new failures	to the given file (or
       FailureRecorder).

   run_suite(properties...)
	 my $num_successful = $runner->run_suite( @properties );
	 if ($num_successful ==	@properties) {
	     # celebrate most jubilantly!
	 }

       Checks a	suite of properties, sending the results of each property
       checked to "STDOUT" in a	form that is compatible	with
       Test::Harness::TAP.  For	example:

	 1..5
	 ok 1 -	Property->new disallows	use of 'tcon' in bindings
	 ok 2 -	magic Property syntax disallows	use of 'tcon' in bindings
	 ok 3 -	exceptions are caught and reported as failures
	 ok 4 -	pre-flight check catches new w/	no args
	 ok 5 -	pre-flight check catches unbalanced arguments list

       By default, labeling statistics and counterexamples (if any) are
       included	in the output if the TestRunner's "verbose" property is	true.
       You may override	the default by passing the "verbose" named parameter
       after all of the	properties in the argument list:

	 my $num_successes = $runner->run_suite( @properties,
						 verbose => 1 );
	 my $num_failed	= @properties -	$num_successes;

HELPER OBJECTS
       There are two kinds of objects that TestRunner uses as helpers.
       Neither is meant	to be created by you.  Rather, a TestRunner will
       create them on your behalf when they are	needed.

       The objects are described in the	following subsections.

   Test::LectroTest::TestRunner::results
	 my $results = $runner->run( $a_property );
	 print "Property name: ", $results->name, ": ";
	 print $results->success ? "Winner!" : "Loser!";

       This is the object that you get back from "run".	 It contains all of
       the information available about the outcome of a	property check and
       provides	the following methods:

       success
	   Boolean value:  True	if the property	checked	out successfully;
	   false otherwise.

       summary
	   Returns a one line summary of the property-check outcome.  It does
	   not end with	a newline.  Example:

	     ok	1 - Property->new disallows use	of 'tcon' in bindings

       details
	   Returns all relevant	information about the property-check outcome
	   as a	series of lines.  The last line	is terminated with a newline.
	   The details are identical to	the summary (except for	the
	   terminating newline)	unless label frequencies are present or	a
	   counterexample is present, in which case the	details	will have
	   these extras	(the summary does not).	 Example:

	     1..1
	     not ok 1 -	'my_sqrt meets defn of sqrt' falsified in 1 attempts
	     # Counterexample:
	     # $x = '0.546384454460178';

       name
	   Returns the name of the property to which the results pertain.

       number
	   The number assigned to the property that was	checked.

       counterexample
	   Returns the counterexample that "broke" the code being tested, if
	   there is one.  Otherwise, returns an	empty string.  If any notes
	   have	been attached to the failing trial, they will be included.

       labels
	   Label counts.  If any labels	were applied to	trials during the
	   property check, this	value will be a	reference to a hash mapping
	   each	combination of labels to the count of trials that had that
	   particular combination.  Otherwise, it will be undefined.

	   Note	that each trial	is counted only	once --	for the	most-specific
	   combination of labels that was applied to it.  For example,
	   consider the	following labeling logic:

	     Property {
	       ##[ x <-	Int ]##
	       $tcon->label("negative")	if $x <	0;
	       $tcon->label("odd")	if $x %	2;
	       1;
	     },	name =>	"negative/odd labeling example";

	   For a particular trial, if x	was 2 (positive	and even), the trial
	   would receive no labels.  If	x was 3	(positive and odd), the	trial
	   would be labeled "odd".  If x was -2	(negative and even), the trial
	   would be labeled "negative".	 If x was -3 (negative and odd), the
	   trial would be labeled "negative & odd".

       label_frequencies
	   Returns a string containing a line-by-line accounting of labels
	   applied during the series of	trials:

	     print $results->label_frequencies;

	   The corresponding output looks like this:

	     25% negative
	     25% negative & odd
	     25% odd

	   If no labels	were applied, an empty string is returned.

       exception
	   Returns the text of the exception or	error that caused the series
	   of trials to	be aborted, if the trials were aborted because an
	   exception or	error was intercepted by LectroTest.  Otherwise,
	   returns an empty string.

       attempts
	   Returns the count of	trials performed.

       incomplete
	   In the event	that the series	of trials was halted before it was
	   completed (such as when the retry count was exhausted), this	method
	   will	return the reason.  Otherwise, it returns an empty string.

	   Note	that a series of trials	is complete if a counterexample	was
	   found.

   Test::LectroTest::TestRunner::testcontroller
       During a	live property-check trial, the variable	$tcon is available to
       your Properties.	 It lets you label the current trial or	request	that
       it be re-tried with new inputs.

       The following methods are available.

       retry
	       Property	{
		 ##[ x <- Int ]##
		 return	$tcon->retry if	$x == 0;
	       }, ... ;

	   Stops the current trial and tells the TestRunner to re-try it with
	   new inputs.	Typically used to reject a particular case of inputs
	   that	doesn't	make for a good	or valid test.	While not required,
	   you will probably want to call "$tcon->retry" as part of a "return"
	   statement to	prevent	further	execution of your property's logic,
	   the results of which	will be	thrown out should it run to
	   completion.

	   The return value of "$tcon->retry" is itself	meaningless; it	is the
	   side-effect of calling it that causes the current trial to be
	   thrown out and re-tried.

       label(string)
	       Property	{
		 ##[ x <- Int ]##
		 $tcon->label("negative") if $x	< 0;
		 $tcon->label("odd")	  if $x	% 2;
	       }, ... ;

	   Applies a label to the current trial.  At the end of	the trial, all
	   of the labels are gathered together,	and the	trial is dropped into
	   a bucket bearing the	combined label.	 See the discussion of
	   "labels" for	more.

       trivial
	       Property	{
		 ##[ x <- Int ]##
		 $tcon->trivial	if $x == 0;
	       }, ... ;

	   Applies the label "trivial" to the current trial.  It is identical
	   to calling "label" with "trivial" as	the argument.

       note(string...)
	       Property	{
		 ##[ s <- String( charset=>"A-Za-z0-9" ) ]##
		 my $s_enc     = encode($s);
		 my $s_enc_dec = decode($s_enc);
		 $tcon->note("s_enc	= $s_enc",
			     "s_enc_dec	= $s_enc_dec");
		 $s eq $s_enc_dec;
	       }, name => "decode is encode's inverse" ;

	   Adds	a note (or notes) to the current trial.	 In the	event that the
	   trial fails,	these notes will be emitted as part of the
	   counterexample.  For	example:

	       1..1
	       not ok 1	- property 'decode is encode's inverse'	\
		   falsified in	68 attempts
	       #     Counterexample:
	       #     $s	= "0";
	       #     Notes:
	       #     $s_enc	= "";
	       #     $s_enc_dec	= "";

	   Notes can help you debug your code when something goes wrong.  Use
	   them	as debugging hints to yourself.	 For example, you can use
	   notes to record the output of each stage of a multi-stage test.
	   That	way, if	the test fails,	you can	see what happened in each
	   stage without having	to plug	the counterexample into	your code
	   under a debugger.

	   If you want to include complicated values or	data structures	in
	   your	notes, see the "dump" method, next, which may be more
	   appropriate.

       dump(value, name)
	       Property	{
		 ##[ s <- String ]##
		 my $s_enc     = encode($s);
		 my $s_enc_dec = decode($s_enc);
		 $tcon->dump($s_enc, "s_enc");
		 $tcon->dump($s_enc_dec, "s_enc_dec");
		 $s eq $s_enc_dec;
	       }, name => "decode is encode's inverse" ;

	   Adds	a note to the current trial in which the given value is
	   dumped.  The	value will be dumped via Data::Dumper and thus may be
	   complex and contain weird control characters	and so on.  If you
	   supply a name, it will be used to name the dumped value.  Returns
	   value as its	result.

	   In the event	that the trial fails, the note (and any	others)	will
	   be emitted as part of the counterexample.

	   See "note" above for	more.

SEE ALSO
       Test::LectroTest::Property explains in detail what you can put inside
       of your property	specifications.

       Test::LectroTest::RegressionTesting explains how	to test	for
       regressions and corner cases using LectroTest.

       Test::Harness:TAP documents the Test Anything Protocol, Perl's simple
       text-based interface between testing modules such as Test::LectroTest
       and the test harness Test::Harness.

AUTHOR
       Tom Moertel (tom@moertel.com)

INSPIRATION
       The LectroTest project was inspired by Haskell's	QuickCheck module by
       Koen Claessen and John Hughes:
       http://www.cs.chalmers.se/~rjmh/QuickCheck/.

COPYRIGHT and LICENSE
       Copyright (c) 2004-13 by	Thomas G Moertel.  All rights reserved.

       This program is free software; you can redistribute it and/or modify it
       under the same terms as Perl itself.

perl v5.32.0			  2013-05-16   Test::LectroTest::TestRunner(3)

NAME | VERSION | SYNOPSIS | DESCRIPTION | METHODS | HELPER OBJECTS | SEE ALSO | AUTHOR | INSPIRATION | COPYRIGHT and LICENSE

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=Test::LectroTest::TestRunner&sektion=3&manpath=FreeBSD+12.2-RELEASE+and+Ports>

home | help