Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
ATF-TEST-CASE(4)	 BSD Kernel Interfaces Manual	      ATF-TEST-CASE(4)

NAME
     atf-test-case -- generic description of test cases

DESCRIPTION
     A test case is a piece of code that stress-tests a	specific feature of
     the software.  This feature is typically self-contained enough, either in
     the amount	of code	that implements	it or in the general idea that de-
     scribes it, to warrant its	independent testing.  Given this, test cases
     are very fine-grained, but	they attempt to	group similar smaller tests
     which are semantically related.

     A test case is defined by three components	regardless of the language it
     is	implemented in:	a header, a body and a cleanup routine.	 The header
     is, basically, a declarative piece	of code	that defines several proper-
     ties to describe what the test case does and how it behaves.  In other
     words: it defines the test	case's meta-data, further described in the
     Meta-data section.	 The body is the test case itself.  It executes	all
     actions needed to reproduce the test, and checks for failures.  This body
     is	only executed if the abstract conditions specified by the header are
     met.  The cleanup routine is a piece of code always executed after	the
     body, regardless of the exit status of the	test case.  It can be used to
     undo side-effects of the test case.  Note that almost all side-effects of
     a test case are automatically cleaned up by the library; this is ex-
     plained in	more detail in the rest	of this	document.

     It	is extremely important to keep the separation between a	test case's
     header and	body well-defined, because the header is always	parsed,
     whereas the body is only executed when the	conditions defined in the
     header are	met and	when the user specifies	that test case.

     At	last, test cases are always contained into test	programs.  The test
     programs act as a front-end to them, providing a consistent interface to
     the user and several APIs to ease their implementation.

   Results
     Upon termination, a test case reports a status and, optionally, a textual
     reason describing why the test reported such status.  The caller must en-
     sure that the test	case really performed the task that its	status de-
     scribes, as the test program may be bogus and therefore providing a mis-
     leading result, e.g., providing a result that indicates success but the
     error code	of the program says otherwise.

     The possible exit status of a test	case are one of	the following:

     expected_death	 The test case expects to terminate abruptly.

     expected_exit	 The test case expects to exit cleanly.

     expected_failure	 The test case expects to exit with a controller fa-
			 tal/non-fatal failure.	 If this happens, the test
			 program exits with a success error code.

     expected_signal	 The test case expects to receive a signal that	makes
			 it terminate.

     expected_timeout	 The test case expects to execute for longer than its
			 timeout.

     passed		 The test case was executed successfully.  The test
			 program exits with a success error code.

     skipped		 The test case could not be executed because some pre-
			 conditions were not met.  This	is not a failure be-
			 cause it can typically	be resolved by adjusting the
			 system	to meet	the necessary conditions.  This	is al-
			 ways accompanied by a reason, a message describing
			 why the test was skipped.  The	test program exits
			 with a	success	error code.

     failed		 An error appeared during the execution	of the test
			 case.	This is	always accompanied by a	reason,	a mes-
			 sage describing why the test failed.  The test	pro-
			 gram exits with a failure error code.

     The usefulness of the `expected_*'	results	comes when writing test	cases
     that verify known failures	caused,	in general, due	to programming errors
     (aka bugs).  Whenever the faulty condition	that the `expected_*' result
     is	trying to cover	is fixed, then the test	case will be reported as
     `failed' and the developer	will have to adjust it to match	its new	condi-
     tion.

     It	is important to	note that all `expected_*' results are only provided
     as	a hint to the caller; the caller must verify that the test case	did
     actually terminate	as the expected	condition says.

   Input/output
     Test cases	are free to print whatever they	want to	their stdout(4)	and
     stderr(4) file descriptors.  They are, in fact, encouraged	to print sta-
     tus information as	they execute to	keep the user informed of their	ac-
     tions.  This is specially important for long test cases.

     Test cases	will log their results to an auxiliary file, which is then
     collected by the test program they	are contained in.  The developer need
     not care about this as long as he uses the	correct	APIs to	implement the
     test cases.

     The standard input	of the test cases is unconditionally connected to
     `/dev/zero'.

   Meta-data
     The following list	describes all meta-data	properties interpreted inter-
     nally by ATF.  You	are free to define new properties in your test cases
     and use them as you wish, but non-standard	properties must	be prefixed by
     `X-'.

     descr		Type: textual.	Required.

			A brief	textual	description of the test	case's pur-
			pose.  Will be shown to	the user in reports.  Also
			good for documentation purposes.

     has.cleanup	Type: boolean.	Optional.

			If set to true,	specifies that the test	case has a
			cleanup	routine	that has to be executed	by the runtime
			engine during the cleanup phase	of the execution.
			This property is automatically set by the framework
			when defining a	test case with a cleanup routine, so
			it should never	be set by hand.

     ident		Type: textual.	Required.

			The test case's	identifier.  Must be unique inside the
			test program and should	be short but descriptive.

     require.arch	Type: textual.	Optional.

			A whitespace separated list of architectures that the
			test case can be run under without causing errors due
			to an architecture mismatch.

     require.config	Type: textual.	Optional.

			A whitespace separated list of configuration variables
			that must be defined to	execute	the test case.	If any
			of the required	variables is not defined, the test
			case is	skipped.

     require.diskspace	Type: integer.	Optional.  Specifies the minimum
			amount of available disk space needed by the test.
			The value can have a size suffix such as `K', `M', `G'
			or `T' to make the amount of bytes easier to type and
			read.

     require.files	Type: textual.	Optional.

			A whitespace separated list of files that must be
			present	to execute the test case.  The names of	these
			files must be absolute paths.  If any of the required
			files is not found, the	test case is skipped.

     require.machine	Type: textual.	Optional.

			A whitespace separated list of machine types that the
			test case can be run under without causing errors due
			to a machine type mismatch.

     require.memory	Type: integer.	Optional.  Specifies the minimum
			amount of physical memory needed by the	test.  The
			value can have a size suffix such as `K', `M', `G' or
			`T' to make the	amount of bytes	easier to type and
			read.

     require.progs	Type: textual.	Optional.

			A whitespace separated list of programs	that must be
			present	to execute the test case.  These can be	given
			as plain names,	in which case they are looked in the
			user's PATH, or	as absolute paths.  If any of the re-
			quired programs	is not found, the test case is
			skipped.

     require.user	Type: textual.	Optional.

			The required privileges	to execute the test case.  Can
			be one of `root' or `unprivileged'.

			If the test case is running as a regular user and this
			property is `root', the	test case is skipped.

			If the test case is running as root and	this property
			is `unprivileged', the runtime engine will automati-
			cally drop the privileges if the `unprivileged-user'
			configuration property is set; otherwise the test case
			is skipped.

     timeout		Type: integral.	 Optional; defaults to `300'.

			Specifies the maximum amount of	time the test case can
			run.  This is particularly useful because some tests
			can stall either because they are incorrectly coded or
			because	they trigger an	anomalous behavior of the pro-
			gram.  It is not acceptable for	these tests to stall
			the whole execution of the test	program.

			Can optionally be set to zero, in which	case the test
			case has no run-time limit.  This is discouraged.

   Environment
     Every time	a test case is executed, several environment variables are
     cleared or	reseted	to sane	values to ensure they do not make the test
     fail due to unexpected conditions.	 These variables are:

     HOME	    Set	to the work directory's	path.

     LANG	    Undefined.

     LC_ALL	    Undefined.

     LC_COLLATE	    Undefined.

     LC_CTYPE	    Undefined.

     LC_MESSAGES    Undefined.

     LC_MONETARY    Undefined.

     LC_NUMERIC	    Undefined.

     LC_TIME	    Undefined.

     TZ		    Hardcoded to `UTC'.

   Work	directories
     The test program always creates a temporary directory and switches	to it
     before running the	test case's body.  This	way the	test case is free to
     modify its	current	directory as it	wishes,	and the	runtime	engine will be
     able to clean it up later on in a safe way, removing any traces of	its
     execution from the	system.	 To do so, the runtime engine will perform a
     recursive removal of the work directory without crossing mount points; if
     a mount point is found, the file system will be unmounted (if possible).

   File	creation mode mask (umask)
     Test cases	are always executed with a file	creation mode mask (umask) of
     `0022'.  The test case's code is free to change this during execution.

SEE ALSO
     atf-test-program(1)

BSD				 March 6, 2017				   BSD

NAME | DESCRIPTION | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=atf-test-case&sektion=4&manpath=FreeBSD+12.0-RELEASE+and+Ports>

home | help