This chapter illustrates how the basic features of ATAC and
In this tutorial we illustrate how the basic features of ATAC and
The word counting program takes as arguments an optional list of files and an optional combination of the flags -l, -w, and -c, each respectively indicating whether to count lines, words, or characters within the argument files. By default, all input is read from standard input and all lines, words, and characters are counted. The source code and sample input for the wordcount program are contained in the files main.c, wc.c, Makefile, input1, input2, and input3. The complete source listings of the first three files appear in Appendix A, Platform Specific Information. These files are also installed with ATAC so you may execute these commands as you read this tutorial. To copy these files, create a new directory, cd to it, and copy the contents of the directory in which the tutorial files are installed into the new directory.
Before using ATAC, check that the word counting program compiles and runs on a sample input. To create your executable program, type the appropriate make or nmake command to build on your system. If you are unsure which command to use, refer to Appendix A, Platform Specific Information.
The output should indicate that the source (.c) files are processed and an executable called
wordcount(.exe) is created.
prompt:> wordcount input1The file input1 contains the following line (the first character is a tab):
test input file 1The output of wordcount should look like this:
1 4 19 input1 1 4 19 totalNow you are ready to use ATAC. Remove the previously created object files and the executable file. One way to do this is to use the clean command appropriate for your setup. Recompile the wordcount program with ATAC. Refer to Appendix A, Platform Specific Information for assistance determining these exact commands and for an approximate example of the output you will see.
As discussed in Section 5.2.2, Integrating with Makefiles, ATAC is easily integrated with existing makefiles. Again, if you do not wish to use (n)make, you may compile the program under ATAC by entering the appropriate compile command at your system prompt. If errors are encountered during compilation refer to Section 5.4, Compilation and Link Errors.
Notice that in addition to creating the .o or .obj files and the executable file, ATAC has
created main.atac and wc.atac. ATAC creates a .atac file for each .c file it compiles. Each
.atac file contains a list of what needs to be covered when testing its corresponding .c file.
This static coverage information is used later during test analysis.
ATAC by entering the following command:
prompt:> xsuds *.atacFigure 2-1 shows the main
|
![]() |
The scroll bar to the left of the source window displays a thumbnail sketch of the entire file.
Note that there are no white regions in the scroll bar at this point as we have not run the
instrumented program (the executable compiled with ATAC) on any inputs, so no blocks
in the file have been covered yet.
ATAC also provides keyboard shortcuts.
Pressing the Up or Down arrow key will move the text up or down one line at a time. The
PageUp and PageDown keys scroll up and down the source file one page at a time,
respectively. The Home key scrolls to the beginning of the file, whereas the End key goes
to the end of the file.
|
prompt:> wordcount input1 (wordcount.1)
This test should produce the same output as that produced by the version of wordcount
compiled without ATAC as shown earlier.
ATAC to incorporate the dynamic information from this trace file into its display,
click with the left mouse button on the ``File'' button in the top button bar. This will cause
the file menu to pop up. Select the ``open trace file...'' entry in the menu. This will open a
dialog box as shown in Figure 2-3.
(The Windows dialog box looks slightly different.)
|
|
prompt:> wordcount -x input1 (wordcount.2)
It should produce an appropriate error message. Note that
ATAC highlights the covered
and uncovered blocks in the source code and prioritizes them into an order in which you
should try to cover them. It does not construct the tests or determine what inputs are needed
to cover the uncovered code. Constructing the tests is the role of the tester. It does, however,
simplify the tester's job by guiding him or her into creating a small set of high-efficiency,
high-leverage test cases that yield high coverage quickly.
ATAC has highlighted the ``Update'' button in the top button bar, as shown in
Figure 2-6, to alert you to this fact.
ATAC continuously monitors the specified trace files
to see if any new coverage information has been added to them. If so, it highlights the
``Update'' button to indicate this to you. You may choose to click on this button now to
update the display with the coverage information from the test case you just ran, or you may
choose to wait until you have run several test cases.
|
|
The scroll bar in Figure 2-7 indicates that there are very few colored blocks left in the file. Recall, however, that the program consists of two files, main.c and wc.c, and so far we have only been looking at main.c. To look at the overall picture involving both files, click on the ``Summary'' button in the top button bar.
|
Note that each coverage bar is actually made up of two bars, one contained inside the other. The length of the outer bar represents the maximum possible (100%) coverage for the corresponding file and that of the inner bar represents the actual coverage attained so far for that file. As the actual coverage increases, the length of the inner bar increases accordingly. When it reaches 100%, the inner bar spans the entire length of the outer bar as in the case of the wc.c bar in Figure 2-8.
The relative lengths of the outer bars of individual files represent the relative sizes of
various files in terms of, in this case, blocks. As wc.c has about one-third the number of
blocks compared to main.c, the outer bar of the former is about one-third the size of the
latter.
|
|
|
To go back to the summary window, click on the ``Summary'' button in the top button bar. This causes the summary window of Figure 2-8 to be redisplayed with one exception: the wc.c label now appears selected instead of the main.c label indicating that wc.c was the last file selected.
As mentioned earlier, the main.c as well as the ``total'' entries indicate that we have not achieved 100% block coverage yet. Although complete block coverage does not guarantee that a set of tests will reveal all errors, testing is certainly incomplete if there are blocks of code that are not exercised by any test. So click on the main.c label to go back to the main.c source display, as shown previously in Figure 2-7.
Click near the red region in the scroll bar. This will bring the corresponding part of the file into the source window, as shown in Figure 2-12. The red block will be executed when the program reads its input from the standard input instead of a file. Execute the following command which copies the contents of input1 to the standard input of wordcount:
prompt:> wordcount < input1 (wordcount.3)
This should produce the following output:
1 4 19
|
|
prompt:> wordcount nosuchfile (wordcount.4) prompt:> wordcount -wlc input1 (wordcount.5)
The former test case should produce an error message indicating that the file could not be
found and the latter test case should produce the same output as wordcount.1, as shown
earlier.
|
|
Click on the ``by-type'' button. This will show the coverages achieved so far using various coverage measures, as shown in Figure 2-16. The first two entries indicate that the five tests you ran have covered all 3 of 3 function entries and 51 of 51 blocks in all source files. The next three entries provide the coverage status for other coverage measures known as decision, c-use, and p-use (see Section 3.3, What Does ATAC Do? for an explanation of these measures). Note that none of these measures have reached a 100% coverage status yet. Let us now try to raise the decision coverage to 100%.
|
A decision is a conditional branch from one block to another. As can be seen from the coverage summary in Figure 2-16, it is possible that a set of tests will cover all blocks in a program without covering some of the decisions. In this example, 30 of the total 35 decisions have been covered. In order to determine what additional test cases are needed to cover the remaining five decisions, click on the ``decision'' button in the third row. Figure 2-17 shows the resulting display.
|
|
The scroll bar shows that there are several expressions highlighted in red. Use the bottom (or top) arrow in the scroll bar to scroll up (or down) the source window by a few lines so the highlighted expression controlling the do-while loop becomes visible in the source window. Then click on the highlighted expression to pop up the list of all branches originating there, as shown in Figure 2-19.
|
To cover the true branch of the loop expression, you must invoke wordcount with more than one input file. Execute the following command to do this:
prompt:> wordcount input1 input2 (wordcount.6)It should produce the following output:
1 4 19 input1 2 8 38 input2 3 12 57 totalClick on the highlighted ``Update'' button to read the coverage information from the above test case. Figure 2-20 shows the relevant part of the updated display.
|
|
The scroll bar now indicates that there are still three nonwhite conditional expressions towards the bottom of the file and one towards the top of the file. Click near the top of the scroll bar to bring the corresponding text in the source window. Then click on the highlighted switch expression to show the corresponding branch list,as shown in Figure 2-21. The branch list indicates that four of the five possible branches of the switch statement have already been covered. The remaining branch can be covered by invoking the wordcount program with a ``-?'' option. The following command accomplishes this5:
prompt:> wordcount -? (wordcount.7)This test case should print an appropriate usage message. Click on the ``Update'' button to confirm that it has, indeed, covered the desired branch of the switch statement. Then close the branch list window by clicking on the ``dismiss'' entry.
So far we have invoked wordcount with options that have caused it to print all three - line, word, and character - counts. We have never invoked it to print only one or two of these counts. Examining the remaining uncovered decisions reveals that we should invoke the program with only one of the three options, -l, -w, and -c, in order to cover these uncovered decisions. The following two commands achieve this:
prompt:> wordcount -l input1 (wordcount.8) prompt:> wordcount -w input1 (wordcount.9)They should produce appropriate line and word counts, respectively, for the file, input1. Click on the ``Update'' button and you will see that all decisions in the file have been covered. Now go back to the summary window to check the overall coverage status by clicking on the ``Summary'' button and selecting the ``by-type'' coverage option. Figure 2-22 shows the new coverage summary.
|
Click on the ``c_use'' button in the summary display. Figure 2-23 shows the resulting display. A c-use display for a file highlights all the definitions of, or the assignments to, the variables involved in all c-uses in the file. If a c-use assignment is highlighted in white, it means all c-uses originating at that assignment have been covered. If, on the other hand, it is highlighted in a nonwhite color it means that there is at least one c-use originating at this assignment that has not been covered yet. For example, the white highlighting of the assignment to the variable p in the loop initialization of the for loop in Figure 2-23 indicates that all c-uses involving this assignment have already been covered. The assignment of the variable doline in the first switch branch inside the for loop, on the other hand, is highlighted in red. This means there are one or more c-uses of this assignment that have not been covered yet. To see which ones, click on the assignment statement. Figure 2-24 shows the resulting display after scrolling down the file so the highlighted c-uses become visible in the source window.
|
|
prompt:> wordcount -l < input1 (wordcount.10)
Updating the display with the ``Update'' button confirms that the uncovered c-use of doline
has been covered.
Figure 2-23:
prompt:> wordcount -w < input1 (wordcount.11) prompt:> wordcount -c < input1 (wordcount.12)
prompt:> wordcount -l nosuchfile (wordcount.13) prompt:> wordcount -lx input1 (wordcount.14) prompt:> wordcount input1 nosuchfile (wordcount.15)
Update the display with the ``Update'' button. All c-uses in the file have now been covered. Display the overall coverage summary by clicking on the ``Summary'' button and selecting the ``by-type'' coverage option, as shown in Figure 2-25. The c-use summary indicates that only 89 of the total 92 c-uses have been covered yet. But all c-use assignments in main.c were highlighted in white indicating that all c-uses in that file have been covered. This means the remaining three uncovered c-uses must be in the file wc.c.
|
To switch to wc.c display, click on the ``File'' button in the top button bar and select the ``wc.c'' entry from the resulting file menu. Figure 2-26 shows the c-use display for wc.c.
|
prompt:> wordcount empty (wordcount.16)
|
Figure 2-26 |
Update the display using the ``Update'' button to check that the uncovered c-use is covered by the last test case. Click on the ``Summary'' button and select the ``c-use'' entry to go back to the c-use definitions display, as shown in Figure 2-28. Note that besides covering the desired c-use involving the variable nl, the last test case also covered the analogous c-uses involving the variables nw and nc. Figure 2-29 shows the new coverage summary. As the c-use entry in the summary indicates, you have now achieved a 100% c-use coverage.
|
|
The fifth row of the coverage summary in Figure 2-29 indicates the current p-use coverage status. A p-use, or a predicate variable use, is a combination of an assignment to a variable, a subsequent use of that variable in a conditional expression, and a particular branch originating at that conditional expression (see Section 3.3, What Does ATAC Do?). Thus a p-use is like a c-use except that the variable use is in a branch originating at a conditional expression. Note that 30 of the total 31 p-uses have already been covered.
To see the only remaining uncovered p-use, click on the ``p_use''
button in the summary display. Figure 2-30
shows the resulting display.
Like the c-use display, the p-use display
highlights all the definitions of, or assignments to, the variables involved in all p-uses in
the file. If a p-use assignment is highlighted in white, it means all p-uses originating at that
assignment have been covered. Four of the five highlighted assignments6 in Figure 2-30 are
highlighted in white. The nonwhite color of the remaining assignment indicates that there
is at least one p-use originating at that assignment that is yet to be covered. To see all the
p-uses of this assignment, click on the highlighted assignment expression. Figure 2-31
shows the resulting display.
It highlights all conditional expressions that use the variable
assigned by the assignment in question. In this case there is only one such conditional
expression, highlighted in red. Note that the assignment in question has also been
highlighted in a different color for your reference. Recall that a p-use involves an
assignment, a conditional expression, and a particular branch originating at the conditional
expression. So far, we have only seen the former two elements of the remaining uncovered
p-use. To see the last element, click on the highlighted conditional expression. A window
containing a list of branches originating there pops up, as shown in Figure 2-32.
prompt:> wordcount input3 (wordcount.17)
Click on the ``Update'' button. Figure 2-33 shows the updated branch list indicating that the p-use we were trying to cover was indeed covered.
Note that the false entry in the branch list is not highlighted at all (Figure 2-32 and Figure 2-33) either in white or in a nonwhite color. This is because the corresponding p-use is an infeasible p-use -- it is impossible to cover it by any test case. The assignment involved assigns the value, OUT, to the variable, state. The conditional expression involved checks to see if state has the value OUT. Whenever the value examined by the latter is that assigned by the former, the conditional expression will evaluate to true. Thus it is impossible to cover the corresponding p-use involving the false branch. ATAC automatically detects many infeasible decisions, c-uses and p-uses and ignores them. It cannot, however, detect all such decisions, c-uses or p-uses.7
Click on the ``dismiss'' entry to close the branch list. Then click on the ``Summary'' button
in the top button bar and select the ``by-type'' entry. Figure 2-34 shows the resulting
coverage summary.
All coverage criteria measured by ATAC are now covered. From
ATAC's point of view, these 17 tests are a completely adequate test of wordcount. Of
course, all we have done is create a set of tests that will thoroughly test the program. You
must check that the program actually passed the tests. This may be done while using
ATAC or after recompiling the program with the standard compiler.
|
There is no guarantee that a program which has passed a completely adequate set of tests
has no errors.8 However, in addition to producing test sets that reveal errors, the use of
ATAC and ATAC to achieve high coverage places the source code under intensive
scrutiny which also tends to reveal errors. A complete test set combined with the effort to
create such a test set is very effective at revealing errors. For large programs, it may require
extensive testing to achieve 100% block, decision, c-use, and p-use coverage as we did for
the wordcount program. In practice it may be necessary to settle for less than 100%
coverage.
To quit
ATAC click on the ``File'' button in the top button bar, then select ``exit''.
2
Although the red region, in this case, appears to consist of a single block, it is a sequence of two contiguous basic
blocks, as the first one of the two is a function call. A function call, in general, may never return, e.g., if it invokes
exit under certain conditions. Thus a function call breaks the ``single-entry-single-exit'' property of a basic block
and results in the start of a new basic block at a statement immediately following the function call.
ATAC with the -nosticky option to make its behavior similar to that of a pulldown
menu. In that case, a branch list window will remain popped up as long as you keep the mouse button
pressed. It will be automatically closed when you release the button.
7
No program can automatically detect all infeasible decisions, c-uses or p-uses as the general problem of determining
if a decision, a c-use or a p-use is infeasible is an unsolvable problem.