Skip site navigation (1)Skip section navigation (2)

FreeBSD Manual Pages

  
 
  

home | help
KYUA-TEST(1)		  BSD General Commands Manual		  KYUA-TEST(1)

NAME
     kyua test -- Runs tests

SYNOPSIS
     kyua test [--build-root path] [--kyuafile file] [--results-file file]
	  [test_filter1	.. test_filterN]

DESCRIPTION
     The kyua test command loads a test	suite definition from a	kyuafile(5),
     runs the tests defined in it, and records the results into	a new results
     file.  By default,	all tests in the test suite are	executed but the op-
     tional arguments to kyua test can be used to select which test programs
     or	test cases to run.  These are filters and are described	below in Test
     filters.

     Every test	executed by kyua test is run under a controlled	environment as
     described in Test isolation.

     The following subcommand options are recognized:

     --build-root path
	 Specifies the build root in which to find the test programs refer-
	 enced by the Kyuafile,	if different from the Kyuafile's directory.
	 See Build directories below for more information.

     --kyuafile	path, -k path
	 Specifies the Kyuafile	to process.  Defaults to a Kyuafile file in
	 the current directory.

     --results-file path, -s path
	 Specifies the results file to create.	Defaults to `LATEST', which
	 causes	kyua test to automatically generate a new results file for the
	 test run.

	 The following values are accepted:

	 `NEW'
	     Requests the automatic generation of a new	results	file name
	     based on the test suite being run and the current time.

	 Explicit file name (aka everything else)
	     Store the results file where indicated.

	 See Results files for more details.

     You can later inspect the results of the test run in more detail by using
     kyua-report(1) or you can execute a single	test case with debugging func-
     tionality by using	kyua-debug(1).

   Build directories
     Build directories (or object directories, target directories, product di-
     rectories,	etc.) is the concept that allows a developer to	keep the
     source tree clean from build products by asking the build system to place
     such build	products under a separate subtree.

     Most build	systems	today support build directories.  For example, the GNU
     Automake/Autoconf build system exposes such concept when invoked as fol-
     lows:

	   $ cd	my-project-1.0
	   $ mkdir build
	   $ cd	build
	   $ ../configure
	   $ make

     Under such	invocation, all	the results of the build are left in the
     my-project-1.0/build/ subdirectory	while maintaining the contents of
     my-project-1.0/ intact.

     Because build directories are an integral part of most build systems, and
     because they are a	tool that developers use frequently, kyua test sup-
     ports build directories too.  This	manifests in the form of kyua test be-
     ing able to run tests from	build directories while	reading	the (often im-
     mutable) test suite definition from the source tree.

     One important property of build directories is that they follow (or need
     to	follow)	the exact same layout as the source tree.  For example,	con-
     sider the following directory listings:

	   src/Kyuafile
	   src/bin/ls/
	   src/bin/ls/Kyuafile
	   src/bin/ls/ls.c
	   src/bin/ls/ls_test.c
	   src/sbin/su/
	   src/sbin/su/Kyuafile
	   src/sbin/su/su.c
	   src/sbin/su/su_test.c

	   obj/bin/ls/
	   obj/bin/ls/ls*
	   obj/bin/ls/ls_test*
	   obj/sbin/su/
	   obj/sbin/su/su*
	   obj/sbin/su/su_test*

     Note how the directory layout within src/ matches that of obj/.  The src/
     directory contains	only source files and the definition of	the test suite
     (the Kyuafiles), while the	obj/ directory contains	only the binaries gen-
     erated during a build.

     All commands that deal with the workspace support the --build-root	path
     option.  When this	option is provided, the	directory specified by the op-
     tion is considered	to be the root of the build directory.	For example,
     considering our previous fake tree	layout,	we could invoke	kyua test as
     any of the	following:

	   $ kyua test --kyuafile=src/Kyuafile --build-root=obj
	   $ cd	src && kyua test --build-root=../obj

   Results files
     Results files contain, as their name implies, the results of the execu-
     tion of a test suite.  Each test suite executed by	kyua-test(1) generates
     a new results file, and such results files	can be loaded later on by in-
     spection commands such as kyua-report(1) to analyze their contents.

     Results files support identifier-based lookups and	also path name
     lookups.  The differences between the two are described below.

     The default naming	scheme for the results files provides simple support
     for identifier-based lookups and historical recording of test suite runs.
     Each results file is given	an identifier derived from the test suite that
     generated it and the time the test	suite was run.	Kyua can later look up
     results files by these fileds.

     The identifier follows this pattern:

	   <test_suite>.<YYYYMMDD>-<HHMMSS>-<uuuuuu>

     where `test_suite'	is the path to the root	of the test suite that was run
     with all slashes replaced by underscores and `YYYYMMDD-HHMMSS-uuuuuu' is
     a timestamp with microsecond resolution.

     When using	the default naming scheme, results files are stored in the
     ~/.kyua/store/ subdirectory and each file holds a name of the form:

	   ~/.kyua/store/results.<identifier>.db

     Results files are simple SQLite databases with the	schema described in
     the /usr/local/share/kyua/store/schema_v?.sql files.  For details on the
     schema, please refer to the heavily commented SQL file.

   Test	filters
     A test filter is a	string that is used to match test cases	or test	pro-
     grams in a	test suite.  Filters have the following	form:

	   test_program_name[:test_case_name]

     Where `test_program_name' is the name of a	test program or	a subdirectory
     in	the test suite,	and `test_case_name' is	the name of a test case.

   Test	isolation
     The test programs and test	cases run by kyua test are all executed	in a
     deterministic environment.	 This known, clean environment serves to make
     the test execution	as reproducible	as possible and	also to	prevent
     clashes between tests that	may, for example, create auxiliary files with
     overlapping names.

     For plain test programs and for TAP test programs,	the whole test program
     is	run under a single instance of the environment described in this page.
     For ATF test programs (see	atf(7)), each individual test case and test
     cleanup routine are executed in separate environments.

     Process space
	 Each test is executed in an independent processes.  Corollary:	the
	 test can do whatever it wants to the current process (such as modify
	 global	variables) without having to undo such changes.

     Session and process group
	 The test is executed in its own session and its own process group.
	 There is no controlling terminal attached to the session.

	 Should	the test spawn any children, the children should maintain the
	 same session and process group.  Modifying any	of these settings pre-
	 vents kyua test from being able to kill any stray subprocess as part
	 of the	cleanup	phase.	If modifying these settings is necessary, or
	 if any	subprocess started by the test decides to use a	different
	 process group or session, it is the responsibility of the test	to en-
	 sure those subprocesses are forcibly terminated during	cleanup.

     Work directory
	 The test is executed in a temporary directory automatically created
	 by the	runtime	engine.	 Corollary: the	test can write to its current
	 directory without needing to clean any	files and/or directories it
	 creates.  The runtime engine takes care to recursively	delete the
	 temporary directories after the execution of a	test case.  Any	file
	 systems mounted within	the temporary directory	are also unmounted.

     Home directory
	 The HOME environment variable is set to the absolute path of the work
	 directory.

     Umask
	 The value of the umask	is set to 0022.

     Environment
	 The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY,
	 LC_NUMERIC and	LC_TIME	variables are unset.

	 The TZ	variable is set	to `UTC'.

	 The TMPDIR variable is	set to the absolute path of the	work direc-
	 tory.	This is	to prevent the test from mistakenly using a temporary
	 directory outside of the automatically-managed	work directory,	should
	 the test use the mktemp(3) familiy of functions.

     Process limits
	 The maximum soft core size limit is raised to its corresponding hard
	 limit.	 This is a simple, best-effort attempt at allowing tests to
	 dump core for further diagnostic purposes.

     Configuration varibles
	 The test engine may pass run-time configuration variables to the test
	 program via the environment.  The name	of the configuration variable
	 is prefixed with `TEST_ENV_' so that a	configuration variable of the
	 form `foo=bar'	becomes	accessible in the environment as
	 `TEST_ENV_foo=bar'.

EXIT STATUS
     The kyua test command returns 0 if	all executed test cases	pass or	1 if
     any of the	executed test cases fails or if	any of the given test case
     filters does not match any	test case.

     Additional	exit codes may be returned as described	in kyua(1).

EXAMPLES
   Workflow with results files
     Let's say you run the following command twice in a	row:

	   kyua	test -k	/usr/tests/Kyuafile

     The two executions	will generate two files	with names like	these:

	   ~/.kyua/store/results.usr_tests.20140731-150500-196784.db
	   ~/.kyua/store/results.usr_tests.20140731-151730-997451.db

     Taking advantage of the default naming scheme, the	following commands
     would all generate	a report for the results of the	latest execution of
     the test suite:

	   cd /usr/tests && kyua report
	   cd /usr/tests && kyua report	--results-file=LATEST
	   kyua	report --results-file=/usr/tests
	   kyua	report --results-file=usr_tests
	   kyua	report --results-file=usr_tests.20140731-151730-997451

     But it is also possible to	explicitly load	data for older runs or from
     explicitly-named files:

	   kyua	report \
	       --results-file=usr_tests.20140731-150500-196784
	   kyua	report \
	       --results-file=~/.kyua/store/results.usr_tests.20140731-150500-196784.db

SEE ALSO
     kyua(1), kyua-report(1), kyuafile(5)

BSD			       October 13, 2014				   BSD

NAME | SYNOPSIS | DESCRIPTION | EXIT STATUS | EXAMPLES | SEE ALSO

Want to link to this manual page? Use this URL:
<https://www.freebsd.org/cgi/man.cgi?query=kyua-test&sektion=1&manpath=FreeBSD+12.1-RELEASE+and+Ports>

home | help