Contact: Igor Ostapenko <igoro@FreeBSD.org>

The FreeBSD test suite is executed by the kyua(1) utility. Kyua supports parallel execution of tests with kyua -v parallelism=<n> test, however many network tests leverage jail(8) features like VNET(9) and have conflicts with jail naming and network configuration. As a result they are marked with the is_exclusive=true metadata property to prevent them from running at the same time and interfering with each other. It creates a dilemma when a project aims to increase test coverage, but the accumulation of exclusive tests proportionally increases the time required to run them. This, in turn, affects the development process from multiple angles.

Kyua has recently got a change in 15-CURRENT to support a new concept called "execution environment". By default, tests run in the so-called "host" execution environment, where they are executed as before. A test can opt-in to use a brand new execution environment, the "jail" one. In this case, kyua creates a jail before running the test, and then executes the test within the jail. That opens up the opportunity to run more tests in parallel due to the extra isolation provided by the jail concept itself, and specifically by the VNET. It depends on hardware and configuration, but there are reports that having the same environment netpfil/pf tests can be run around 4 times faster — a few minutes instead of half an hour.

The following Makefile change is a quick demo of how netpfil/pf tests were switched to run in parallel with jail execution environment:

More details:

This change also brings new sysctl read-only variables, which expose more details about current jail, and may be generally useful:

A hint: the sysctl -n security.jail.children.cur run from prison0 provides the number of all jails in the system.

Further improvements to Kyua, such as requirements definition and automatic resolution, are currently in the design phase. Potentially new metadata properties like required_klds and required_pkgs provide a clue to these topics. Please contact Igor to discuss ideas and use cases that can help shape these upcoming Kyua enhancements.