This chapter provides an overview of ATAC and is recommended reading for first-time users or those who want a summary of ATAC.

3.1 What is ATAC?

3.2 What is Coverage Testing?

• All statements should be executed;
  
  
• All decisions should be evaluated both to true and to false.
  
  

Each specific coverage criterion identifies a number of program constructs to be exercised (covered) by a set of tests. The constructs to be covered are called testable attributes. For instance, in covering all decisions as suggested above, there is one testable attribute for each true branch and one for each false branch in the program. A tester covers them by developing a test set that executes each of these branches. A test set is considered adequate with respect to a given coverage criterion if all testable attributes identified by the criterion are exercised, at least once, by some test within the set.

It is harder to develop an adequate test set for some coverage criteria than for others. Weaker criteria usually require fewer test cases than stronger criteria to obtain completely adequate coverage. However, a test set adequate for a weaker criterion is less likely to reveal an error than a test set adequate for a stronger criterion. For example, it generally requires fewer test cases to ensure that all functions within a program are invoked than to ensure that all statements are executed. However, a test set that executes all program statements tends to test a program more thoroughly than a test set that invokes all functions. This is because it is possible to invoke all functions without executing all of their statements. The converse is not true.

One coverage criterion is stronger than another if, for any program, completely adequate coverage for the first implies completely adequate coverage for the second and, as with statement and function coverage, the converse is not true. If the converse is also true, then the coverage criteria are of equal strength.

In practice, for many coverage criteria, a completely adequate test set is not easy to create for most programs. As a test set is being developed, it exhibits a level of adequacy called a coverage measure - the percentage of testable attributes exercised by its set of tests. As a test set's coverage measure improves, it becomes harder to create test cases that cover the remaining, uncovered testable attributes. In some cases, it may be impossible or impractical to achieve completely adequate coverage. For example, a program may contain code to detect and handle error conditions that are very hard to simulate during testing. The appropriate target coverage measures for any program under test depend on the characteristics and reliability requirements of that program.

3.3 What Does ATAC Do?

Figure 3-1 An approximate partial ordering of ATAC coverage criteria

ATAC can instrument all, or a selected portion, of the files making up a software system. This allows testing to be targeted at a specific portion of the system and makes it possible to incrementally manage the overhead of testing a large system. The coverage measures reported only pertain to that source code which has been instrumented, and any instrumentation added has no effect on non-instrumented code.

3.3.1 Function-Entry, Function-Return, Function-Call and Block Coverage

Block coverage ensures that all basic blocks are executed at least once. A basic block is a sequence of instructions that, except for the last instruction, is free of branches and function calls. So, the instructions in any basic block are either executed all together, or not at all. In C and C++, a block may contain more than one statement if no branching occurs between statements; a statement may contain multiple blocks if branching occurs inside the statement; an expression may contain multiple blocks if branching is implied within the expression (e.g., conditional, logical-and, and logical-or expressions). ATAC begins a new basic block after a function call because it is possible that the function call will not return (e.g. if exit is called within the function). ATAC provides an option to allow basic blocks to span function calls.

Figure 3-2 presents an example of three distinct blocks, as they are displayed in ATAC's character-based interface. Block 1 consists of a logical-expression embedded within a compound conditional-expression; Block 2 consists of an entire conditional expression; Block 3 consists of the entire body of an if-statement. If block coverage is ever less than 100%, then there are program statements that have not been executed by any test. So, achieving completely adequate block coverage ensures that the entire program is at least executed. Completely adequate block coverage implies completely adequate function-entry coverage.

Figure 3-2 An example of three distinct blocks (character-based interface)

3.3.2 Decision Coverage

Figure 3-3 An example of true and false decision paths (character-based display)

3.3.3 C-Use, P-use, and coverage criteria All-Uses Coverage

C-use, p-use, and all-uses coverage criteria are based on both the control flow and data flow of a program. C-use (computational use) coverage ensures that if a variable is defined (assigned a value) and later used within a computation that is not part of a conditional expression, at least one path between this def-use pair is executed. Figure 3-4 presents an example c-use, as displayed by ATAC. Because functions and statements need not define or use any variables, c-use coverage is not comparable to most of the other coverage criteria.

Figure 3-4 An example c-use (character-based display)

Figure 3-5 presents two distinct p-uses, as they are displayed by ATAC. Much like decision coverage, there are true and false execution paths passing through all conditional expressions involved in p-uses, each path corresponding to a distinct testable attribute.

The intuition behind c-use and p-use coverage is, when a variable is assigned a value and that value is later used, a good test set will exercise this relationship. For any use of a variable, this should occur for each assignment that might have given rise to the variable's value. All-uses coverage is the sum of p-use and c-use coverage measures.

Figure 3-5 An example of true and false p-uses (character-based interface)

ATAC consists of the following tools: the ATAC compiler (atac cc on UNIX, atacCL or atacICC on Windows), atac, atactm, and xatac. These are the user-layer components of ATAC invoked by a tester from the command line. In addition, there are other executables and a run-time library required by ATAC that are not of general interest to users. An executable called ataclib (UNIX) or atac_lib (Windows) is invoked by all other ATAC components when they need to locate the ATAC library. This helps minimize the need for users to modify their personal computing environment in order to use ATAC.

Figure 3-6 depicts the key inputs and outputs during software instrumentation, test execution, and coverage analysis. Instrumentation of the software to be tested is performed by the ATAC compiler. The ATAC compiler replaces the standard compiler, while accepting all the same command line options. The ATAC compiler supports separate compilation and can be easily used in conjunction with the make or nmake programs. The ATAC compiler accepts one or more .c source files as input and, for each, computes its testable attributes and instruments it to record an execution trace at run-time. The outputs of the ATAC compiler are a .atac file corresponding to each .c file and an executable program, a.out. All instrumentation is emitted as source code embedded within the software to be tested, and then the standard compiler is invoked to generate a.out (an alternative compiler may be invoked if so desired). Each .atac file acts as a list of all testable attributes that exist within its corresponding .c file.

Figure 3-6 Key inputs and outputs during program instrumentation, test execution and coverage analysis

The atactm tool manipulates the contents of a trace file. Trace files contain coverage information for each test that has been executed. Using atactm, a tester can list, rename, extract, delete, or assign cost to individual tests. ATAC permits the coverage achieved by arbitrary subsets of tests to be compared and computes minimal size and cost subsets of tests achieving the same coverage as the entire test set. A tester may use these tools to manage a test set during testing and later compact it without reducing coverage. This approach reduces the cost of any regression testing to be performed in the future.

Although ATAC provides a large number of options for varying the form and content of the information reported, defaults have been chosen so that few options are required to use the basic features. Complete details of the ATAC commands and options appear later in this guide.

3.5 What Will Using ATAC Cost You?

Using ATAC will also require additional disk space. This need arises from three sources: the increased size of the instrumented executable to be tested, the creation of .atac files during compilation, and the creation of a .trace file during test execution. The instrumented executable is approximately twice the size of a non-optimized, non-instrumented executable program. Each .atac file is at least as large as its corresponding .c file. The size of the .trace file generated during testing is a function of the total number of testable attributes to be covered and the size of the test set. When trace file compression is used, the .trace file grows very little unless a test actually improves coverage, in which case its growth is proportional to what is covered. Omitting trace file compression reduces test execution time but makes the .trace file grow more quickly, thereby costing additional disk space. Uncompressed .trace files can become rather large over time.

3.6 How Does ATAC Fit into the Development Process?

The view that ATAC is a unit testing tool used by individual developers is largely one of practicality, but this need not always be the case. All the software components of an entire product can be instrumented and tested at the same time, if the product is of appropriate size. Also, some projects might designate a team of one or more individuals to coverage test all or part of the source code produced by its developers. Typically, it is most efficient for the author of a piece of software to do the coverage testing. This is because developing test cases that improve coverage is easier if one is very familiar with the source code under test. However, there is some flexibility in how ATAC is used to best satisfy the testing requirements and resource limitations of a given project.

ATAC is not generally perceived as a system testing tool because system test focuses on verifying the behavior of software features, not on exercising constructs within the source code. Furthermore, it is likely that a system is too large and/or system testers have insufficient knowledge of a product's architecture to effectively perform coverage testing. Nevertheless, if the resources are available, it may be beneficial to instrument all or part of a product's source code using ATAC, and then run its system test suite. This provides a means of determining the overall code coverage of the system test suite.