Lib:Torture Chamber
From GNUpdf
Contents |
Overview
Welcome to the Torture Chamber for the GNU PDF Library.
We follow a bottom-up testing strategy. The verification of the library is performed in the following steps:
- Unit testing is performed in order to verify the low-level modules of the library.
- Subsystem testing is performed in order to verify the combination of several subsystems. i.e. to test each library layer.
- System testing is performed in order to verify the whole system. i.e. the GNU PDF Library.
We use check (a testing framework for C) to implement our testing infrastructure. Please read the check manual in order to become familiar with its concepts such as test suite, test case, etc.
The Test Specification Document
The tests for the GNU PDF Library are documented in a document called "Test Specification Document" or TSD. It is available as a texinfo file in the sources distribution in the doc/gnupdf-tsd.texi file.
There is an online version of the TSD in Lib:Test_Specification_Document. It is automatically updated daily from the CVS HEAD.
Unit Testing
We organize unit tests using the following structure:
Test suites are used to collect unit tests for a given module. In turn each test suite contain a collection of test cases. Each test case identifies a function implemented in the module. Several tests can then be defined to test the function capabilities.
The unit tests are stored in the torture/unit/ directory.
Designing Unit Tests
Please keep in mind the following considerations when designing unit tests for the functions of a library module:
For any given function we would like to define positive, negative and stressing unit tests:
- Positive tests are used to test the function with valid input data. The return value (or any documented side effect) of the function should be checked with the expected effect.
- Negative tests are used to test the function with invalid input data.
- Stressing tests are used to test the function with strategic or "interesting" valid input data (such as MAXINT or MININT in a function accepting an integer value).
Many functions are simple enough to only require a single positive test running the function (no parameters). It is important to have this unit test since the execution may fail even for such simple functions.
While designing the unit tests you may find that the API contain errors or that it may be improved in any way. That is fine and it is quite welcomed. You can propose any change of the API in this list.
Test Files
The test files contain collections of unit tests associated with a function. Each test file is named function-name.c, where function-name is the name of the function under testing with underscores replaced with dashes.
For example, the source file containing unit tests for the pdf_stm_new function would be called stm-new.c.
A minimal test file looks like this:
/*
* Include the check library.
*/
#include <check.h>
/*
* START_TEST and END_TEST are macros provided by the check
* library.
*/
START_TEST(FUNCTION_NAME_NNN)
{
/* Check a condition. One of several types of check
available in the check library.
*/
fail_if(0 == 1);
}
END_TEST
/*
* Provide this function to gather all the tests together.
*/
TCase* test_FUNCTION_NAME (void)
{
TCase* tc = tcase_create ("FUNCTION_NAME");
tcase_add_test(tc, FUNCTION_NAME_NNN);
return tc;
}
Note the test_FUNCTION_NAME function. It is the function called by the test driver in order to perform all the tests implemented in the test file.
Test Suite Files
Once you add a new test file, that defines a testcase for some module function, you need to integrate it into the test suite that defines the module itself.
Each module directory (such as torture/unit/base/stm) has a test suite definition file such as tsuite-stm.c. The suite definition file defines a function called tsuite_MODULE-NAME that return a Suite * object.
New test cases are added using the suite_add_tcase function:
extern TCase *my_new_tcase (void);
Suite *
tsuite_mymodule ()
{
Suite *s;
...
suite_add_tcase (s, my_new_tcase ());
...
return s;
}
The Runtests Program
The runtests program is the driver of the testing procedure. It is located in the torture/unit/ directory.
In order to integrate new test files into the runtests program the source files of the test files should be added to the value of the 'TEST_FILES' variable in 'torture/unit/Makefile.am'. For example:
TEST_FILES = base/stm/pdf-create-file-stm.c \
base/OTHER-MODULE/OTHER-TEST-FILE.c
Running the Unit Tests
The runtests program will run all the defined unit tests of the library. It can be invoked (and it is the preferred way) by typing
$ make check
in the console.
The test driver display several messages to the standard output and also dump a logfile named ut.log with details about the test execution.
Using gdb to debug check tests
The check testing framework uses fork calls in order to create the processes used to run the single tests. This makes possible to caught unexpected process terminations such as a segmentation fault or a division by zero.
Sometimes we want to debug those failure conditions using gdb. Unfortunately the GNU debugger cannot caught the unexpected termination of the child processess.
The check implementor foreseen this and provides a workaround: to define the CK_FORK variable to "no" and launch the debugger.
