TESTSET
Version 1.0
READ.ME

Copyright (c) 1997, Terrence Lambert
All rights reserved.

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.
3. All advertising materials mentioning features or use of this software
   must display the following acknowledgement:
	This product includes software developed by Terrence Lambert.
4. The name of the author may not be used to endorse or promote products
   derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.


1.0	What is TESTSET?

	The testset framework is a framework for automation of user
	space test code for detection of kernel memory leakage in a
	zone allocation environment.

	Testset is a tool for automating kernel software change
	validation for rapid recovery following large scale kernel
	changes.

	This code has been tested and run on FreeBSD, and is derived
	from similar source code by the same author for Solaris and
	SVR4.


2.0	What can TESTSET do?

	Testset operates in three phases:

	o	Setup
	o	Run
	o	Teardown

	In each of these phases, the framework runs user space code
	in order to trigger an error.  Therefore this code is most
	useful when used in order to detect kernel memory leaks
	that result from user space calls into kernel space.  It is
	also possible to implement "side effect leakage" testing
	using "split tests" (see 4.0).

	Because the kernel virtual memory is accessed to examine it
	for leakage, the program must be run as root.  If you are
	attempting to investigate a bug that you believe is related
	to leakage specifically in the non-superuser credential case
	of some kernel code, your test functions should be "setuid"
	and restore to root credentials afterwards (it is recommended
	that this be implemented using POSIX saved ID's).  If your
	entire set of tests expects to be run as non-root, it is
	suggested you encapsulate the save/restore operations in
	common functions which you call on entry to (and exit from)
	each test case.

	If an initialization error occurs, testset will exit with a
	status of '1'.  It is expected that this will be the result
	of a user coding error in the usr_setup() routine (see 2.1),
	or a result of inability to access the kernel virtual memory
	space or symbols within that space (the most common cause of
	this is not running the tool as the root user).

	If a leak is detected, the leak is reported as to which user
	defined test triggered the leak, and which monitored zone or
	zones in which the leak occurred.  In event of a leak being
	detected, no further tests will be run (for fear of aggravating
	the problem to the point of system crash), and testset will
	exit with a status of '2'.

	If no leak is detected, after all tests have completed,
	testset will report "No leaks detected" on stdout, and exit
	with a a status of '0'.

	Because of these status reporting facilities, test suites
	based on the testset framework are suitable for inclusion in
	a series of tests; for example, an OS validation framework.


2.1	Setup

	In the setup phase, the usr_setup function is called with the
	complete program command line arguments.  This allows the user
	to implement variant operations of a single large set of test
	code (for instance, running a single large set of tests with
	and without root credentials in order to test both code paths,
	and so on).

	The usr_setup function provided by the user must match the
	following prototype:

	int usr_setup( int ac, char *av[]);

	The function must return 0 if setup is successful, and 1 (or
	other exit value for the testset itself -- 0..9 are reserved)
	if the initilization fails.  Users might find it useful to
	return a value of 10 or more in order to distinguish between
	testset initialization errors and usr_setup initialization
	errors, or various types of usr_setup errors.

	The purpose of usr_setup is to set up the environment required
	by the actual tests themselve; this may include the creation
	of files (for instance, if the tests involve opening or renaming
	or deleting an existing file), or providing a target directory
	(if the tests are for a particular FS type, ie: a cryptographic
	or network file system), etc..


2.2	Run

	In the run phase, an array of test sets (struct testset) is
	traversed.  In order, the procedure is to:

	o	Obtain baseline values for the monitored memory zones
	o	Call the user test function to operate against the
		environment setup by the usr_setup function
	o	Obtain post-test values for the monitored memory zones,
		and by comparing them with the baseline values, detect
		memory leaks on a per zone basis.

	In the event of a leak detection, the test where the leak was
	detected is identified, and the zone or zones which leaked are
	listed.  The test is aborted following this to prevent further
	corruption of the kernel virtual memory by subsequent tests
	(see 2.0).

	The user test sets are provided as an exported array of testset
	structures.  A testset structure looks like:

	struct testset {
		char	*ts_name;
		void	(*ts_func)();
		struct baselines {
			char	*b_name;
			long	b_inuse;
		} ts_base[ MAX_BASE];
	};

	The components are as follows:

	o	ts_name			Name of test
	o	ts_func			Function to call to perform test
	o	ts_base[]		Baseline array; 1..TS_MAX_BASE elems.
	-		b_name		Symbolic name of zone (ie: "namei")
	-		b_inuse		Baseline established for zone
					(this value is not set by the user)

	The user exports an array of these structures named usr_tests:

		struct testset usr_tests[];
	
	The array is terminated by an element with a NULL-valued ts_name.

	The ts_base is an array of up to MAX_BASE (testset itself is
	compiled with this set to 10) b_name values for monitored
	zones (ie "namei", etc.).  It is terminated by either there
	being a full MAX_BASE initialization elements, or by a ts_base
	element with a NULL-valued b_name.

	EXAMPLE:

	struct testset usr_tests[] = {
		{	"Open Existing File",
			open_existing_file,
			{	{ "namei" },
				{ NULL }
			}
		},
		{ NULL }
	};

	Defines a testset consisting of a single test element named
	"Open Existing File", the test function is open_existing_file,
	and there is one monitored zone, "namei", out of the 10
	possible.

	The ts_func member is expected to meet the following declaration
	prototype:

	void my_test_func( void);

	That is, it is expected to have no overt effect or return value
	other than its effect on the kernel virtual address space, if
	any (an effect there in a monitored zone is a failure: it is
	an indication that a memory leak has been detected).  This
	means that if you expect the possibility of an operation in
	the test failing, you should pre-test the operation in the
	usr_setup (see 2.1) to prevalidate the operations success, and
	fail usr_setup rather than failing the test function.

	It is recommended that the functions usr_setup and usr_teardown,
	and the array usr_tests be the only exported symbols from your
	test set's code.  This means that your local test functions
	for each test case should probably be declared to be static.


2.3	Teardown

	In the teardown phase, the usr_teardown function is called.

	The usr_teardown function must be prepared to deal with any
	failure at any point in testing -- that is, if the first 3
	tests in a list of 5 tests have run, and a leak is detected,
	then the usr_teardown must be able to clean up after the
	usr_setup, regardless that all tests have not run to completion.
	This means that if you had, for example, a "rename_test", you
	would have to be able to clean up the file whether or not the
	rename had actually taken place.


3.0	Using testset

	You can use testset by:

	1)	Write a module exporting usr_setup, usr_teardown, and
		usr_tests, as described in section 2, above.

	2)	Link your module as follows:

		ld -o mytestprog mytestprog.o -lts -lkvm

		The 'ts' library will provide 'main()' for you.

	3)	Run the resulting program as root


4.0	Advanced topics

	The following advanced topis are discussed:

	o	Split tests


4.1	Split tests

	It is sometimes necessary to implement "split tests" to enable
	"side effect leakage" detection.

	This is generally done by saving state over several tests.

	For example, rather than having a test function that operates
	by calling open/fstat/close, you could split it into three
	cases: open, fstat, and close, and open a global fd in the
	open cases, fstat the global fd in the fstat case, and close
	the global fd in the fstat case.


5.0	Bug reports and enhancements

	Send your bug report or enhancement suggestion to:

		Terry Lambert
		terry@lambert.org

	Please use one of the following subjects for quickest response:

		SW: TESTSET BUG
		SW: TESTSET REQ

6.0	End of Document