kyua-atf-interface man page

atf-interface — Description of the ATF test program interface

Description

The interface of ATF test programs is the interface of the test programs linked to the atf-c, atf-c++ and atf-sh libraries provided by ATF.

The ATF interface can be understood as the mechanisms used by test programs to communicate with the runtime engine as well as the assumptions that test programs and test cases can make while running.

A test case is the most basic part of a test suite. A test case is supposed to reproduce one, and only one, scenario. For example: if the item under test was a function, the test case would provide a single set of input parameters to the function and check its output; If the item under test was a binary, the test case would provide a single set of arguments to the program and check its behavior.

Test case parts

Test cases have three parts:

Head
Programmatically defines metadata properties. The head must not perform any other thing than defining such properties. In particular, no testing whatsoever can happen in the head. (Ideally the definition of metadata properties would not happen programmatically.)
Body
The actual test case which performs any desired testing and reports a result. The body is executed by the runtime engine in a deterministic way; see the isolation section below.
Cleanup
An optional cleanup routine. Note that the runtime engine will attempt to clean up the work directory automatically, so this routine should only be provided in cases where the test modifies system-wide state not known by the runtime engine. The cleanup part is executed in the same directory as the body. However, the body and the cleanup parts do not share the same process space; the only way to pass data around from the body to the cleanup is by means of files in the work directory.

Metadata properties

The following test case metadata properties must be exported in the test case list for every test case:

ident
Single-word string. The name of the test case. Must be unique within the test program.

The following test case metadata properties may be exported in the test case list for every test case:

descr
Multi-word string. A textual description for the test case. Usually, providing a descriptive identifier is better than providing a textual description.
has.cleanup
Boolean. Whether the test case defines a cleanup routine or not.
require.arch
Whitespace separated list of the architectures required by the test case. If defined, the test case is skipped unless the host architecture matches any of the values defined in this property.
require.config
Whitespace separated list of configuration variable names. The list of configuration variables that must be defined. The test is skipped if any of these is missing.
require.files
Whitespace separated list of absolute paths to installed files. If any of these files is not found, the test case is skipped.
require.machine
Whitespace separated list of the machine types required by the test case. If defined, the test case is skipped unless the host machine type matches any of the values defined in this property.
require.progs
Whitespace separated list of program names (either absolute names or base names). If any of these programs is not found, the test case is skipped.
require.user
Either ‘root’ or ‘unprivileged’. If ‘root’, the test case must be run as the superuser or otherwise it is skipped. If ‘unprivileged’, the test case must be run as an unprivileged user or else it is skipped.
timeout
Integer. The amount of seconds the test case can run for before it is killed by the runtime engine.

Configuration properties

The following properties may be defined by the runtime engine and are propagated to the test cases:

unprivileged-user
String, optional. Specifies the name of the user under which tests that set ‘require.user=unprivileged’ are executed. The tests's UID is set to the given user and the GID is set to the primary GID of the user. All other groups are dropped.

Results

A test case must always report a result by creating the results file specified through the -r flag. For convenience when running test cases without the runtime engine, this file may point to /dev/stdout or /dev/stderr in which case the file must not be created (because the creation will fail).

Aside from creating the results file, the process in which the test case runs must terminate in particular ways for the test result to be considered valid.

If the test case fails to create the test result, if the test result is created but contains an invalid syntax, or if the termination status of the process does not match the requirements of the test result, the runtime engine marks the test case as ‘broken’. Note that the ‘broken’ state is decided by the runtime engine; a test case cannot report itself as ‘broken’.

The general syntax for the results file is as follows:

<status>[[(int)]: reason]

The following results are allowed:

expected_death
The process is expected to terminate either due to a clean call to exit(3) or due to the reception of a signal. The contents of the file are ‘expected_death: <reason>’. Example: ‘expected_death: Calling libdofoo breaks due to bug xyz’.
expected_exit
The process is expected to terminate cleanly. The contents of the file are ‘expected_exit: <reason>’ if the exit code is irrelevant or ‘expected_exit(<exitcode>): <reason>’ if the process must terminate with a given exit code. Examples: ‘expected_exit: Calling bar exits but it should not’ or ‘expected_exit(123): Calling bar exits with an unexpected code’.
expected_failure
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘expected_failure: <reason>’ Example: ‘expected_failure: 2 + 2 = 3’.
expected_signal
The process is expected to terminate due to the reception of a signal. The contents of the file are ‘expected_signal: <reason>’ if the signal number is irrelevant or ‘expected_signal(<signalno>): <reason>’ if the process must terminate due to a particular signal. Examples: ‘expected_signal: Calling bar crashes’ or ‘expected_signal(1): Calling bar kills ourselves due to unhandled SIGHUP’.
expected_timeout
The process is expected to hang for longer than its timeout metadata property. Only the runtime engine can control this situation because the runtime engine is the one implementing the timeout functionality.
failed
The process must exit cleanly with an EXIT_FAILURE exit code. The contents of the file are ‘failed: <reason>’. Example: ‘failed: Failed on purpose’.
passed
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘passed’.
skipped
The process must exit cleanly with an EXIT_SUCCESS exit code. The contents of the file are ‘skipped: <reason>’. Example: ‘skipped: Skipped because the foo is not present’.

Isolation

The runtime engine attempts to isolate test cases from other test cases in the same test program and from the rest of the system by performing what is called test case isolation.

Whenever the user runs a test program binary by hand (i.e. not through kyua(1)), the test program will print a warning message stating that test case isolation does not work and therefore the program may cause side-effects and/or report invalid values.

The runtime engine must set the __RUNNING_INSIDE_ATF_RUN environment variable to the magic value ‘internal-yes-value’ to tell the test programs that they are being run with isolation enabled.

The test case isolation performs the following:

Process space
Each test case body and cleanup routines are executed in independent processes. Corollary: the test case can do whatever it wants to the current process (such as modifying global variables) without having to undo such changes.
Session and process group
The test case body and cleanup are executed in their own session and their own process group. There is no controlling terminal attached to the session.

Should the test case spawn any children, the children should maintain the same session and process group. Modifying any of these prevents the tester from killing any stray subprocess as part of the cleanup phase. If modifying these is necessary, or if any subprocess started by the test case decides to use a different process group or session, it is the responsibility of the test case to ensure they are forcibly terminated during cleanup.

Work directory
The test case body and its cleanup (if any) are executed in a temporary directory automatically created by the runtime engine. This temporary directory is shared among the body and cleanup parts of a single test case but is completely separate from the temporary directories of other tests. Corollary: the test case body and cleanup routines can write to their current directory without bothering to clean any files and/or directories they create. The runtime engine takes care to recursively delete the temporary directories after the execution of a test case. Any file systems mounted within the temporary directory will be unmounted if possible.
Home directory
The HOME environment variable is set to the absolute path of the work directory.
Umask
The value of the umask is set to 0022.
Environment
The LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, LC_MONETARY, LC_NUMERIC and LC_TIME variables are unset. The TZ variable is set to ‘UTC’.
Process limits
The maximum soft core size limit is raised to its corresponding hard limit. This is a simple, best-effort attempt at allowing test cases to dump core for further diagnostic purposes.

Test programs

A test program is, simply put, a collection of related test cases. The test program can be seen as a command-line dispatcher for the test cases. A test program must provide one or more test cases. If it does not contain any test case, the runtime system will report it as invalid.

Test programs expose their list of test cases in a machine parseable format. The runtime engine obtains the list of test cases to know what tests to run and to know how to set up the environment of each test prior execution. The test program must not do any test when asked to dump its test case list.

The generic syntax to obtain the list of test cases included in a test program is:

<test-program> -l

The list of test cases follows the following format:

LIST ::= HEADER NEWLINE TEST_CASES 
 
HEADER ::= 'Content-Type: application/X-atf-tp; version="1"' 
NEWLINE ::= '\n' 
TEST_CASES ::= TEST_CASE | TEST_CASE NEWLINE TEST_CASES 
 
TEST_CASE ::= IDENT_PROPERTY PROPERTIES 
IDENT_PROPERTY ::= 'ident' DELIM STRING NEWLINE 
DELIM ::= ': ' 
 
PROPERTIES ::= PROPERTY | PROPERTY PROPERTIES 
PROPERTY ::= PROPERTY_NAME DELIM STRING NEWLINE 
PROPERTY_NAME ::= (see below)

An example:

Content-Type: application/X-atf-tp; version="1" 
 
ident: addition 
descr: Tests that the addition function works 
 
ident: subtraction 
descr: Tests that the subtraction function works 
 
ident: remove 
descr: Tests removing files 
require.root: true 
timeout: 50 
has.cleanup: true

The syntax to run a test case body part is:

<test-program> [-r resfile] [-s srcdir] [-v var=value]* <test-case>[:body]

This must run the test case body “as is”, without any attempt of isolating it from the rest of the system. It is the responsibility of the runtime engine to do such isolation.

The runtime engine always passes the path of a nonexistent file to -r, which must be created by the test case; and always passes an absolute path to the -s flag pointing to the directory containing the test program executable.

The runtime engine shall pass any configuration variables it wants through the -v flag, and these can be later inspected by the test case at will.

A note to users: if you run the test case by hand (not through kyua(1) nor atf-run(1)) from the command line, none of the isolation features described in the isolation section apply. This means that the test case can (and probably will) write to the current directory and leave garbage behind. Also, given that the test case is executed without e.g. clearing the environment, the results of the test case may differ from those obtained when running the test case inside the runtime engine. Only use this for debugging purposes (i.e. to run the test case code under GDB).

The syntax to run a test case cleanup part is:

<test-program> [-s srcdir] [-v var=value]* <test-case>:cleanup

This can only be performed if and only if the test case sets the has.cleanup property to true. Otherwise the behavior of executing the cleanup part is undefined.

The same rules for -s and -v apply as to when running the body.

The cleanup part must be executed in the same directory as the body but in a separate process space. The only way for test cases to transfer state (if any) from the body to the cleanup routine is by means of files in the current directory.

The cleanup part does not have to worry about deleting temporary files created in the current directory. The runtime engine does this automatically.

See Also

kyua-test(1), kyuafile(5)

Referenced By

kyua-atf-tester(1), kyuafile(5), kyua-plain-interface(7), kyua-tap-interface(7), kyua-tester(1).

March 11, 2014