kyua-atf-interface man page
atf-interface — Description of the ATF test program interface
The interface of ATF test programs is the interface of the test programs linked to the atf-c, atf-c++ and atf-sh libraries provided by ATF.
The ATF interface can be understood as the mechanisms used by test programs to communicate with the runtime engine as well as the assumptions that test programs and test cases can make while running.
A test case is the most basic part of a test suite. A test case is supposed to reproduce one, and only one, scenario. For example: if the item under test was a function, the test case would provide a single set of input parameters to the function and check its output; If the item under test was a binary, the test case would provide a single set of arguments to the program and check its behavior.
Test case parts
Test cases have three parts:
Programmatically defines metadata properties. The head must not perform any other thing than defining such properties. In particular, no testing whatsoever can happen in the head. (Ideally the definition of metadata properties would not happen programmatically.)
The actual test case which performs any desired testing and reports a result. The body is executed by the runtime engine in a deterministic way; see the isolation section below.
An optional cleanup routine. Note that the runtime engine will attempt to clean up the work directory automatically, so this routine should only be provided in cases where the test modifies system-wide state not known by the runtime engine. The cleanup part is executed in the same directory as the body. However, the body and the cleanup parts do not share the same process space; the only way to pass data around from the body to the cleanup is by means of files in the work directory.
The following test case metadata properties must be exported in the test case list for every test case:
Single-word string. The name of the test case. Must be unique within the test program.
The following test case metadata properties may be exported in the test case list for every test case:
Multi-word string. A textual description for the test case. Usually, providing a descriptive identifier is better than providing a textual description.
Boolean. Whether the test case defines a cleanup routine or not.
Whitespace separated list of the architectures required by the test case. If defined, the test case is skipped unless the host architecture matches any of the values defined in this property.
Whitespace separated list of configuration variable names. The list of configuration variables that must be defined. The test is skipped if any of these is missing.
Whitespace separated list of absolute paths to installed files. If any of these files is not found, the test case is skipped.
Whitespace separated list of the machine types required by the test case. If defined, the test case is skipped unless the host machine type matches any of the values defined in this property.
Whitespace separated list of program names (either absolute names or base names). If any of these programs is not found, the test case is skipped.
Either ‘root’ or ‘unprivileged’. If ‘root’, the test case must be run as the superuser or otherwise it is skipped. If ‘unprivileged’, the test case must be run as an unprivileged user or else it is skipped.
Integer. The amount of seconds the test case can run for before it is killed by the runtime engine.
The following properties may be defined by the runtime engine and are propagated to the test cases:
String, optional. Specifies the name of the user under which tests that set ‘require.user=unprivileged’ are executed. The tests's UID is set to the given user and the GID is set to the primary GID of the user. All other groups are dropped.
A test case must always report a result by creating the results file specified through the
-r flag. For convenience when running test cases without the runtime engine, this file may point to
/dev/stderr in which case the file must not be created (because the creation will fail).
Aside from creating the results file, the process in which the test case runs must terminate in particular ways for the test result to be considered valid.
If the test case fails to create the test result, if the test result is created but contains an invalid syntax, or if the termination status of the process does not match the requirements of the test result, the runtime engine marks the test case as ‘broken’. Note that the ‘broken’ state is decided by the runtime engine; a test case cannot report itself as ‘broken’.
The general syntax for the results file is as follows:
The following results are allowed:
The process is expected to terminate either due to a clean call to exit(3) or due to the reception of a signal. The contents of the file are ‘expected_death: <reason>’. Example: ‘expected_death: Calling libdofoo breaks due to bug xyz’.
The process is expected to terminate cleanly. The contents of the file are ‘expected_exit: <reason>’ if the exit code is irrelevant or ‘expected_exit(<exitcode>): <reason>’ if the process must terminate with a given exit code. Examples: ‘expected_exit: Calling bar exits but it should not’ or ‘expected_exit(123): Calling bar exits with an unexpected code’.
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘expected_failure: <reason>’ Example: ‘expected_failure: 2 + 2 = 3’.
The process is expected to terminate due to the reception of a signal. The contents of the file are ‘expected_signal: <reason>’ if the signal number is irrelevant or ‘expected_signal(<signalno>): <reason>’ if the process must terminate due to a particular signal. Examples: ‘expected_signal: Calling bar crashes’ or ‘expected_signal(1): Calling bar kills ourselves due to unhandled SIGHUP’.
The process is expected to hang for longer than its timeout metadata property. Only the runtime engine can control this situation because the runtime engine is the one implementing the timeout functionality.
The process must exit cleanly with an EXIT_FAILURE exit code. The contents of the file are ‘failed: <reason>’. Example: ‘failed: Failed on purpose’.
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘passed’.
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘skipped: <reason>’. Example: ‘skipped: Skipped because the foo is not present’.
The runtime engine attempts to isolate test cases from other test cases in the same test program and from the rest of the system by performing what is called test case isolation.
Whenever the user runs a test program binary by hand (i.e. not through kyua(1)), the test program will print a warning message stating that test case isolation does not work and therefore the program may cause side-effects and/or report invalid values.
The runtime engine must set the __RUNNING_INSIDE_ATF_RUN environment variable to the magic value ‘internal-yes-value’ to tell the test programs that they are being run with isolation enabled.
The test case isolation performs the following:
- Process space
Each test case body and cleanup routines are executed in independent processes. Corollary: the test case can do whatever it wants to the current process (such as modifying global variables) without having to undo such changes.
- Session and process group
The test case body and cleanup are executed in their own session and their own process group. There is no controlling terminal attached to the session.
Should the test case spawn any children, the children should maintain the same session and process group. Modifying any of these prevents the tester from killing any stray subprocess as part of the cleanup phase. If modifying these is necessary, or if any subprocess started by the test case decides to use a different process group or session, it is the responsibility of the test case to ensure they are forcibly terminated during cleanup.
- Work directory
The test case body and its cleanup (if any) are executed in a temporary directory automatically created by the runtime engine. This temporary directory is shared among the body and cleanup parts of a single test case but is completely separate from the temporary directories of other tests. Corollary: the test case body and cleanup routines can write to their current directory without bothering to clean any files and/or directories they create. The runtime engine takes care to recursively delete the temporary directories after the execution of a test case. Any file systems mounted within the temporary directory will be unmounted if possible.
- Home directory
The HOME environment variable is set to the absolute path of the work directory.
The value of the umask is set to 0022.
The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC and LC_TIME variables are unset. The TZ variable is set to ‘UTC’.
- Process limits
The maximum soft core size limit is raised to its corresponding hard limit. This is a simple, best-effort attempt at allowing test cases to dump core for further diagnostic purposes.
A test program is, simply put, a collection of related test cases. The test program can be seen as a command-line dispatcher for the test cases. A test program must provide one or more test cases. If it does not contain any test case, the runtime system will report it as invalid.
Test programs expose their list of test cases in a machine parseable format. The runtime engine obtains the list of test cases to know what tests to run and to know how to set up the environment of each test prior execution. The test program must not do any test when asked to dump its test case list.
The generic syntax to obtain the list of test cases included in a test program is:
The list of test cases follows the following format:
LIST ::= HEADER NEWLINE TEST_CASES HEADER ::= 'Content-Type: application/X-atf-tp; version="1"' NEWLINE ::= '\n' TEST_CASES ::= TEST_CASE | TEST_CASE NEWLINE TEST_CASES TEST_CASE ::= IDENT_PROPERTY PROPERTIES IDENT_PROPERTY ::= 'ident' DELIM STRING NEWLINE DELIM ::= ': ' PROPERTIES ::= PROPERTY | PROPERTY PROPERTIES PROPERTY ::= PROPERTY_NAME DELIM STRING NEWLINE PROPERTY_NAME ::= (see below)
Content-Type: application/X-atf-tp; version="1" ident: addition descr: Tests that the addition function works ident: subtraction descr: Tests that the subtraction function works ident: remove descr: Tests removing files require.root: true timeout: 50 has.cleanup: true
The syntax to run a test case body part is:
<test-program> [-r resfile] [-s srcdir] [-v var=value]* <test-case>[:body]
This must run the test case body “as is”, without any attempt of isolating it from the rest of the system. It is the responsibility of the runtime engine to do such isolation.
The runtime engine always passes the path of a nonexistent file to
-r, which must be created by the test case; and always passes an absolute path to the
-s flag pointing to the directory containing the test program executable.
The runtime engine shall pass any configuration variables it wants through the
-v flag, and these can be later inspected by the test case at will.
A note to users: if you run the test case by hand (not through kyua(1) nor atf-run(1)) from the command line, none of the isolation features described in the isolation section apply. This means that the test case can (and probably will) write to the current directory and leave garbage behind. Also, given that the test case is executed without e.g. clearing the environment, the results of the test case may differ from those obtained when running the test case inside the runtime engine. Only use this for debugging purposes (i.e. to run the test case code under GDB).
The syntax to run a test case cleanup part is:
<test-program> [-s srcdir] [-v var=value]* <test-case>:cleanup
This can only be performed if and only if the test case sets the has.cleanup property to true. Otherwise the behavior of executing the cleanup part is undefined.
The same rules for
-v apply as to when running the body.
The cleanup part must be executed in the same directory as the body but in a separate process space. The only way for test cases to transfer state (if any) from the body to the cleanup routine is by means of files in the current directory.
The cleanup part does not have to worry about deleting temporary files created in the current directory. The runtime engine does this automatically.
kyua-atf-tester(1), kyuafile(5), kyua-plain-interface(7), kyua-tap-interface(7), kyua-tester(1).