Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
aegis -Test(1)		    General Commands Manual		aegis -Test(1)

NAME
	aegis test - run tests

SYNOPSIS
	aegis -Test [ option...	 ][ name=value ][ file-name...	]
	aegis -Test -INDependent [ option...  ][ name=value ][ file-name...  ]
	aegis -Test -List [ option...  ]
	aegis -Test -Help

DESCRIPTION
	The aegis -Test	command	is used	to run tests.  If no files are named,
	all relevant tests are run.  By	default	both automatic and manual
	tests are run.

	You may	name directories on the	command	line, and all relevant tests
	in that	directory tree in the change will be run.  It is an error if
	there are no relevant tests.

	Each architecture must be tested separately.  This is because there
	may be subtle problems that are	only revealed on some architectures.
	Some projects may also have different code for different architec-
	tures.

	The status of the last test run	is remembered so that tests are	not
	run if there is	no need.  (This	does not apply to -REGression tests,
	unfortunately.)	 Tests must be re-run if the test previously failed,
	if the test file has changed, if there has been	a build, and for each
	architecture.

   name=value
	You can	add name=value pairs to	the command line, these	will be	passed
	unchanged to the test command.	Usually	on the end of the command
	line, but this can be changed in the project configuration file.

	The -force option results in an	implicit force=1 variable being	added
	to the list of variable	assignments, and thus added to the end of the
	command.  This is of most use when using the batch_test_command	filed
	of the project configuration file.

	This may initially look	like a development process end-run, allowing
	test scripts to	be written so that they	give all the right answers
	without	actually doing anything.  You have always been able to do this
	with environment variables, so this isn't anything new.

	It is possible to get all of the variable assignments to turn into en-
	vironment variables by putting $var at the start of the	command, be-
	fore the name of the shell, rather than	at the default location	at the
	end of the command.

   File	Name Interpretation
	The aegis program will attempt to determine the	project	file names
	from the file names given on the command line.	All file names are
	stored within aegis projects as	relative to the	root of	the baseline
	directory tree.	 The development directory and the integration direc-
	tory are shadows of this baseline directory, and so these relative
	names apply here, too.	Files named on the command line	are first con-
	verted to absolute paths if necessary.	They are then compared with
	the baseline path, the development directory path, and the integration
	directory path,	to determine a baseline-relative name.	It is an error
	if the file named is outside one of these directory trees.

	The -BAse_RElative option may be used to cause relative	filenames to
	be interpreted as relative to the baseline path; absolute filenames
	will still be compared with the	various	paths in order to determine a
	baseline-relative name.

	The relative_filename_preference in the	user configuration file	may be
	used to	modify this default behavior.  See aeuconf(5) for more infor-
	mation.

TEST PROCESS
	Each change is required	to be accompanied by tests, and	those tests
	are required to	be run against the built development directory,	and
	they must pass.	 This ensures that new functionality is	accompanied by
	tests to verify	its correctness, and bug fixes are accompanied by
	tests which confirm that the bug has been fixed.

   Regression Tests
	Tests are treated as any other source file, and	are maintained in the
	baseline and history with all other source files.  The tests which
	must accompany every change accumulate in the project baseline,	pro-
	viding a definition of correct function	for the	baseline.  These accu-
	mulated	tests may be executed using an "aegis -REGression" command, to
	verify that the	project	will not "regress" as a	result of a change.

   Baseline Tests
	Bug fixes are required to have their tests fail	against	the project
	baseline (in contrast to the development directory).  This ensures
	that the test actually demonstrates the	bug in the baseline, as	well
	as demonstrating that it is fixed by the change.  New functionality
	trivially fails	against	the baseline, and so aegis does	not attempt to
	guess if a test	is a bug fix test or new functionality test, it	simply
	requires tests to fail against the baseline.

	This requirement applies both to new tests being created by a change
	and also to tests which	have been copied into a	change for modifica-
	tion.

   Reviewing Tests
	Reviewers may be confident that	aegis has enforced the test require-
	ments; that a change must have tests, that the change must build, that
	the tests pass against the development directory, and that the tests
	fail against the baseline.  These conditions are enforced by aede(1)
	and the	change will not	be advanced to the being reviewed state	until
	these conditions are met.  Reviewers should thus review	tests for com-
	pleteness of coverage of the code in the change, and insensitivity to
	changes	in the execution environment (e.g. not date sensitive).	 Re-
	viewers	should also use	"aegis -list change_details" to	verify that a
	change does or does not	have testing exemptions.

   Exemptions
	Various	test exemptions	may be granted by project administrators, see
	aepa(1)	and aepattr(5) for more	information.  Copying tests into a
	change,	or adding new tests to a change, may cancel those exemptions.

TEST COMMAND CONFIGURATION
	The command used to execute tests is defined by	the test_command field
	in the project configuration file (see aepconf(5) for more informa-
	tion), this defaults to	using the Bourne shell if not set.  The	cur-
	rent directory will be the top of the appropriate directory tree.  If
	tests require temporary	files, they should create them in /tmp,	as a
	test cannot expect to have write permission in the current directory.

	If you want to use a more sophisticated	test engine, rather than a
	simple shell script, but this test engine does not return result codes
	suitable for use with aegis, you could wrap it in a shell script which
	re-writes the exit status into the values aegis	expects.  You could
	also achieve the same results by writing a more	complex	test_command
	in the project config file.

	It is also possible to write test commands which are able to test more
	than one file at once.	This is	controlled by the batch_test_command
	field of the project config file.  In this case, the ${output} substi-
	tution indicates the name of a file the	test command must create, in
	aetest(5) format, to contain the results of the	tests run.  This is
	often used on systems with multiple CPUs or the	ability	to distribute
	jobs across several computers on a network.

   Substitutions
	All of the aesub(5) substitutions are available	in the test commands.
	Some of	them are of particular note:

	ARCHitecture
		This substitution is replaced by the name of the architecture
		to be tested.

	Search_Path
		This substitution is replaced by a colon separated list	of ab-
		solute paths to	search when looking for	test support files.

	Search_Path_Executable
		This substitution is replaced by a colon separated list	of ab-
		solute paths to	search when looking for	executable support
		files (library files and sub-commands).

	Most of	the time $Search_Path_Executable are exactly the same.	How-
	ever, during "aegis -t -bl" they will be different, with $Seach_Path
	starting at the	development directory (the test	being run) and
	$Seach_Path_Executable starting	at the baseline	(the executable	being
	run).

   Test	Result Codes
	As each	test is	run (via the test_command field	in the project config
	file), aegis determines	whether	the test succeeded or failed by	look-
	ing at its exit	status.	 This exit status is mostly as expected	for
	UNIX commands.

	Success
	    A test should exit 0 to indicate success, i.e. that	the specific
	    function under test	worked as expected.

	Failure
	    A test should exit 1 to indicate failure, i.e. that	the specific
	    function under test	did not	work as	expected.

	No Result
	    A test should exit 2 to indicate no	result,	i.e. that the specific
	    function under test	could not be exercised because something else
	    went wrong.	 For example, running out of disk space	when creating
	    the	test input files in the	/tmp directory.

	Skipped
	    A test should exit 77 to indicate that it was skipped.  This is
	    usually to do with the current architecture	not being meaningful.
	    Whenever possible, use "No Result" instead.	 (The value was	chosen
	    for	compatibility with other test systems.)

	Actually, any exit code	other than 0, 1	or 77 will be interpreted as
	"no result".  However, always using 0, 1, 2 or 77 means	that if	a new
	result code is required	by a later release of Aegis your existing
	tests will continue to work.

TEST CORRELATIONS
	The "aegis -Test -SUGgest" command may be used to have aegis suggest
	suitable regression tests for your change, based on the	source files
	in your	change.	 This automatically focuses testing effort to relevant
	tests, reducing	the number of regression tests necessary to be confi-
	dent that you have not introduced a bug.

	The test correlations are generated by the "aegis -Integrate_Pass"
	command, which associates each test in the change with each source
	file in	the change.  Thus, each	source file accumulates	a list of
	tests which have been associated with it in the	past.  This is not as
	exact as code coverage analysis, but is	a reasonable approximation in
	practice.

	The aecp(1) and	aenf(1)	commands are used to associate files with a
	change.	 While they do not actively perform the	association, these are
	the files used by aeipass(1) and aet(1)	to determine which source
	files are associated with which	tests.

   Test	Correlation Accuracy
	Assuming that the testing correlations are accurate and	that the tests
	are evenly distributed across the function space, there	will be	a less
	than 1/number chance that a relevant test has not been run by the
	"aegis -Test -SUGgest number" command.	A small	amount of noise	is
	added to the test weighting, so	that unexpected	things are sometimes
	tested,	and the	same tests are not run every time.

	Test correlation accuracy can be improved by ensuring that:

	o Each change should be	strongly focused, with no gratuitous file in-
	  clusions.  This avoids spurious correlations.

	o Each item of new functionality should	be added in an individual
	  change, rather than several together.	 This strongly correlates
	  tests	with functionality.

	o Each bug should be fixed in an individual change, rather than	sev-
	  eral together.  This strongly	correlates tests with functionality.

	o Test correlations will be lost if files are moved.  This is because
	  correlations are by name.

	The best way for tests to correlate accurately with source files is
	when a change contains a test and exactly those	files relating to the
	functionality under test.  Too many spurious files will	weaken the
	usefulness of the testing correlations.

OPTIONS
	The following options are understood:

	-AUTOmatic
		This option may	be used	to specify automatic tests.  Automatic
		tests require no human assistance.

	-BaseLine
		This option may	be used	to specify that	the project baseline
		is the subject of the command.

	-BAse_RElative
		This option may	be used	to cause relative filenames to be con-
		sidered	relative to the	base of	the source tree.  See aeu-
		conf(5)	for the	corresponding user preference.

	-CUrrent_RElative
		This option may	be used	to cause relative filenames to be con-
		sidered	relative to the	current	directory.  This is usually
		the default.  See aeuconf(5) for the corresponding user	pref-
		erence.

	-Change	number
		This option may	be used	to specify a particular	change within
		a project.  See	aegis(1) for a complete	description of this
		option.

	-FOrce	This option may	be used	to specify that	all tests should be
		run, even if the status	of the last test run indicates that
		there is no need to run	a specific test.

	-Help
		This option may	be used	to obtain more information about how
		to use the aegis program.

	-INDependent
		This option is used to specify that the	test is	to be run in-
		dependent of any particular change.  If	no tests are named,
		all tests in the baseline will be run.

	-List
		This option may	be used	to obtain a list of suitable subjects
		for this command.  The list may	be more	general	than expected.

	-MANual	This option may	be used	to specify manual tests.  Manual tests
		require	some human intervention,  e.g.:	confirmation of	some
		screen behavior	(X11, for instance), or	some user action, "un-
		plug ethernet cable now".

	-Not_Logging
		This option may	be used	to disable the automatic logging of
		output and errors to a file.  This is often useful when	sev-
		eral aegis commands are	combined in a shell script.

	-PErsevere
		This option may	be used	to specify that	all tests should be
		run, even if some fail.	 Defaults to the user's	perse-
		vere_preference	if not specified, see aeuconf(5) for more in-
		formation.

	-No_PErsevere
		This option may	be used	to specify that	the test run should
		stop after the first failure.  Defaults	to the user's perse-
		vere_preference	if not specified, see aeuconf(5) for more in-
		formation.

	-Project name
		This option may	be used	to select the project of interest.
		When no	-Project option	is specified, the AEGIS_PROJECT	envi-
		ronment	variable is consulted.	If that	does not exist,	the
		user's $HOME/.aegisrc file is examined for a default project
		field (see aeuconf(5) for more information).  If that does not
		exist, when the	user is	only working on	changes	within a sin-
		gle project, the project name defaults to that project.	 Oth-
		erwise,	it is an error.

	-PROGress
		This option may	be used	to specify that	progress messages
		should be issued before	each test run or before	each batch
		test run in case batch_test_command field specified in project
		config file (see aeuconf(5) for	more information).

	-No_PROGress
		This option may	be used	to specify that	progress messages
		should be suppressed.  This is the default.

	-REGression
		This option is used to specify that the	regression test	suite
		is to be run.  The regression test suite consists of all tests
		in the baseline	which do not appear in the change.  It is an
		error if there are no regression tests.	 You may not name
		tests on the command line when using the -REGression option.
		You may	name individual	tests to be run	on the command line,
		without	using the -REGression option; if they are not part of
		the change, the	tests of the same name in the baseline will be
		run.

	-SUGgest [ number ]
		The "aegis -Integrate_Pass" command collects test correlation
		statistics when	changes	are integrated.	 This option may be
		used to	request	that aegis suggest which tests should be run,
		using these testing correlations.  If no number	is specified,
		10 tests will be suggested.  This option implies the -REGres-
		sion option.

	-SUGgest_Limit minutes
		This option may	be used	to limit the number of tests to	a cer-
		tain number of minutes.	 They will be run from most relevant
		to least relevant.

	-SUGgest_Noise number
		This option may	be used	to control the amount of noise in-
		jected into the	test selection performed by the	-SUGgest op-
		tion.  The number is a percentage of noise to be injected.
		Defaults to 10 if not specified.  The injection	of noise en-
		sures that a variety of	tests are run on subsequent runs, and
		also some from left-field as a sanity check.

	-TERse
		This option may	be used	to cause listings to produce the bare
		minimum	of information.	 It is usually useful for shell
		scripts.

	-Verbose
		This option may	be used	to cause aegis to produce more output.
		By default aegis only produces output on errors.  When used
		with the -List option this option causes column	headings to be
		added.

	-Wait	This option may	be used	to require Aegis commands to wait for
		access locks, if they cannot be	obtained immediately.  De-
		faults to the user's lock_wait_preference if not specified,
		see aeuconf(5) for more	information.

	-No_Wait
		This option may	be used	to require Aegis commands to emit a
		fatal error if access locks cannot be obtained immediately.
		Defaults to the	user's lock_wait_preference if not specified,
		see aeuconf(5) for more	information.

	See also aegis(1) for options common to	all aegis commands.

	All options may	be abbreviated;	the abbreviation is documented as the
	upper case letters, all	lower case letters and underscores (_) are op-
	tional.	 You must use consecutive sequences of optional	letters.

	All options are	case insensitive, you may type them in upper case or
	lower case or a	combination of both, case is not important.

	For example: the arguments "-project", "-PROJ" and "-p"	are all	inter-
	preted to mean the -Project option.  The argument "-prj" will not be
	understood, because consecutive	optional characters were not supplied.

	Options	and other command line arguments may be	mixed arbitrarily on
	the command line, after	the function selectors.

	The GNU	long option names are understood.  Since all option names for
	aegis are long,	this means ignoring the	extra leading '-'.  The	"--op-
	tion=value" convention is also understood.

RECOMMENDED ALIAS
	The recommended	alias for this command is
	csh%	alias aet 'aegis -t \!*	-v'
	sh$	aet(){aegis -t "$@" -v}

ERRORS
	It is an error if the change is	not in one of the being	developed or
	being integrated states.
	It is an error if the change is	not assigned to	the current user.
	It is an error if your have no relevant	tests and no relevant exemp-
	tion.

EXIT STATUS
	The aegis command will exit with a status of 1 on any error.  The
	aegis command will only	exit with a status of 0	if there are no	er-
	rors.

ENVIRONMENT VARIABLES
	See aegis(1) for a list	of environment variables which may affect this
	command.  See aepconf(5) for the project configuration file's
	project_specific field for how to set environment variables for	all
	commands executed by Aegis.

SEE ALSO
	aeb(1)	build a	change

	aeca(1)	modify the attributes of a change

	aedb(1)	begin development of a change

	aeib(1)	begin integration of a change

	aent(1)	add a new test to a change

	aecp(1)	copy an	existing test into a change

	aepconf(5)
		project	configuration file format

	aeuconf(5)
		user configuration file	format

COPYRIGHT
	aegis version 4.25.D510
	Copyright (C) 1991, 1992, 1993,	1994, 1995, 1996, 1997,	1998, 1999,
	2000, 2001, 2002, 2003,	2004, 2005, 2006, 2007,	2008, 2009, 2010,
	2011, 2012 Peter Miller

	The aegis program comes	with ABSOLUTELY	NO WARRANTY; for details use
	the 'aegis -VERSion License' command.  This is free software and you
	are welcome to redistribute it under certain conditions; for details
	use the	'aegis -VERSion	License' command.

AUTHOR
	Peter Miller   E-Mail:	 pmiller@opensource.org.au
	/\/\*		  WWW:	 http://miller.emu.id.au/pmiller/

Reference Manual		     Aegis			aegis -Test(1)

NAME | SYNOPSIS | DESCRIPTION | TEST PROCESS | TEST COMMAND CONFIGURATION | TEST CORRELATIONS | OPTIONS | RECOMMENDED ALIAS | ERRORS | EXIT STATUS | ENVIRONMENT VARIABLES | SEE ALSO | COPYRIGHT | AUTHOR

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=aet&sektion=1&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help