Chapter 2

ATAC: A Tutorial

This chapter illustrates how the basic features of ATAC and ATAC can be used in reporting code coverage and identifying uncovered source code.

In this tutorial we illustrate how the basic features of ATAC and ATAC can be used in testing by way of a running example. ATAC is used to test wordcount, a program that counts the number of lines, words, and/or characters given to it as input. Within this chapter, general terminology is used. See Appendix A, Platform Specific Information, if you need help determining exactly what to type, and to see expected output samples.

The word counting program takes as arguments an optional list of files and an optional combination of the flags -l, -w, and -c, each respectively indicating whether to count lines, words, or characters within the argument files. By default, all input is read from standard input and all lines, words, and characters are counted. The source code and sample input for the wordcount program are contained in the files main.c, wc.c, Makefile, input1, input2, and input3. The complete source listings of the first three files appear in Appendix A, Platform Specific Information. These files are also installed with ATAC so you may execute these commands as you read this tutorial. To copy these files, create a new directory, cd to it, and copy the contents of the directory in which the tutorial files are installed into the new directory.

Before using ATAC, check that the word counting program compiles and runs on a sample input. To create your executable program, type the appropriate make or nmake command to build on your system. If you are unsure which command to use, refer to Appendix A, Platform Specific Information.

The output should indicate that the source (.c) files are processed and an executable called wordcount(.exe) is created.

An alternative way to build the executable is to run your compile command including all the source files in the directory and specifying or renaming wordcount(.exe) as your output file name.

Once wordcount has been built, run it against a sample input:

	prompt:> wordcount input1						

		test input file 1

		1	4	19	input1
		1	4	19	total

As discussed in Section 5.2.2, Integrating with Makefiles, ATAC is easily integrated with existing makefiles. Again, if you do not wish to use (n)make, you may compile the program under ATAC by entering the appropriate compile command at your system prompt. If errors are encountered during compilation refer to Section 5.4, Compilation and Link Errors.

Notice that in addition to creating the .o or .obj files and the executable file, ATAC has created main.atac and wc.atac. ATAC creates a .atac file for each .c file it compiles. Each .atac file contains a list of what needs to be covered when testing its corresponding .c file. This static coverage information is used later during test analysis.

Now invoke ATAC by entering the following command:

	prompt:> xsuds *.atac 

Figure 2-1 The initial display of the main ATAC window

The scroll bar to the left of the source window displays a thumbnail sketch of the entire file. Note that there are no white regions in the scroll bar at this point as we have not run the instrumented program (the executable compiled with ATAC) on any inputs, so no blocks in the file have been covered yet.

The scroll bar is very useful in quickly locating where the red blocks, or the ``hot spots,'' in the file are. Clicking with the left mouse button at any spot in the scroll bar brings the corresponding region of the file into the source window. You may also use the arrows at the top or the bottom of the scroll bar to scroll up or down the source file a few lines at a time. You may also drag the mouse up or down the scroll bar with the left mouse button pressed to rapidly scroll up or down the file. ATAC also provides keyboard shortcuts. Pressing the Up or Down arrow key will move the text up or down one line at a time. The PageUp and PageDown keys scroll up and down the source file one page at a time, respectively. The Home key scrolls to the beginning of the file, whereas the End key goes to the end of the file.

The scroll bar indicates that there is a red spot towards the bottom of the file². Click on or near the red spot so that part of the file becomes visible in the source window. Figure 2-2 shows the resulting display.

Figure 2-2 The source display showing the red blocks or ``hot spots''

 	prompt:> wordcount input1   (wordcount.1) 

This test should produce the same output as that produced by the version of wordcount compiled without ATAC as shown earlier.

Note that in addition to the expected output, running this test case has created an execution trace file called wordcount.trace. This file contains dynamic coverage information used in test analysis. Subsequent tests will cause additional dynamic information to be logged to the same file.

To tell ATAC to incorporate the dynamic information from this trace file into its display, click with the left mouse button on the ``File'' button in the top button bar. This will cause the file menu to pop up. Select the ``open trace file...'' entry in the menu. This will open a dialog box as shown in Figure 2-3. (The Windows dialog box looks slightly different.)

Figure 2-3 The trace file dialog box for UNIX

Note that both the previously red blocks, along with several others, have turned in color to white indicating that they were, indeed, covered by the test case you just ran. The scroll bar also indicates that several other blocks not currently visible in the source window were covered as well. Also note that the ``hot spot'' has now shifted to another statement in the file. ATAC reassigns colors to all uncovered blocks each time it incorporates new dynamic information from a trace file.

Figure 2-4 The source display after executing wordcount.1

 	prompt:> wordcount -x input1   (wordcount.2) 

It should produce an appropriate error message. Note that ATAC highlights the covered and uncovered blocks in the source code and prioritizes them into an order in which you should try to cover them. It does not construct the tests or determine what inputs are needed to cover the uncovered code. Constructing the tests is the role of the tester. It does, however, simplify the tester's job by guiding him or her into creating a small set of high-efficiency, high-leverage test cases that yield high coverage quickly.

Figure 2-5 The source display showing the new ``hot spot''

Running wordcount.2 causes its coverage information to be added to the trace file. Note that ATAC has highlighted the ``Update'' button in the top button bar, as shown in Figure 2-6, to alert you to this fact. ATAC continuously monitors the specified trace files to see if any new coverage information has been added to them. If so, it highlights the ``Update'' button to indicate this to you. You may choose to click on this button now to update the display with the coverage information from the test case you just ran, or you may choose to wait until you have run several test cases.

Figure 2-6 The highlighted Update button

Figure 2-7 The main.c source display after executing wordcount.2

The scroll bar in Figure 2-7 indicates that there are very few colored blocks left in the file. Recall, however, that the program consists of two files, main.c and wc.c, and so far we have only been looking at main.c. To look at the overall picture involving both files, click on the ``Summary'' button in the top button bar. ATAC displays the per file block coverage summary, as shown in Figure 2-8.<sup>3</sup> The summary window shows that the two tests you have run so far have covered 28 of the total 38 blocks in main.c and all 13 of the 13 blocks in wc.c. Overall, they have covered 41 of the 51 blocks, as indicated by the ``total'' entry towards the bottom of the display. The bars on the right display the coverages in terms of percentages. The top two bars indicate that the tests you have run so far have covered 73.7% of the blocks in main.c and 100% of the blocks in wc.c, respectively. The bottom bar indicates that they have covered 80.4% of the total number of blocks.

Figure 2-8 The coverage summary by-file after executing wordcount.2

Note that each coverage bar is actually made up of two bars, one contained inside the other. The length of the outer bar represents the maximum possible (100%) coverage for the corresponding file and that of the inner bar represents the actual coverage attained so far for that file. As the actual coverage increases, the length of the inner bar increases accordingly. When it reaches 100%, the inner bar spans the entire length of the outer bar as in the case of the wc.c bar in Figure 2-8.

The relative lengths of the outer bars of individual files represent the relative sizes of various files in terms of, in this case, blocks. As wc.c has about one-third the number of blocks compared to main.c, the outer bar of the former is about one-third the size of the latter.

If you want to see the summary with respect to each type, click on ``by-type'' in the middle button bar. The result is shown in Figure 2-9. Similarly, click on ``by-function'' for the summary with respect to each function as shown in Figure 2-10.

Figure 2-9 The coverage summary by-type after executing wordcount.2	Figure 2-10 The coverage summary by-function after executing wordcount.2

You may click on a file name in the summary window as shown in Figure 2-8 to see the source display of the corresponding file. For now, click on the wc.c label in that window to see its source display. Figure 2-11 shows the resulting display. As expected, every block in the file is highlighted in white as each one of them has been covered.

Figure 2-11 The wc.c source display after executing wordcount.2

To go back to the summary window, click on the ``Summary'' button in the top button bar. This causes the summary window of Figure 2-8 to be redisplayed with one exception: the wc.c label now appears selected instead of the main.c label indicating that wc.c was the last file selected.

As mentioned earlier, the main.c as well as the ``total'' entries indicate that we have not achieved 100% block coverage yet. Although complete block coverage does not guarantee that a set of tests will reveal all errors, testing is certainly incomplete if there are blocks of code that are not exercised by any test. So click on the main.c label to go back to the main.c source display, as shown previously in Figure 2-7.

Click near the red region in the scroll bar. This will bring the corresponding part of the file into the source window, as shown in Figure 2-12. The red block will be executed when the program reads its input from the standard input instead of a file. Execute the following command which copies the contents of input1 to the standard input of wordcount:

 	prompt:> wordcount < input1   (wordcount.3) 

This should produce the following output:

		1	4	19

Figure 2-12 The main.c source display showing the new ``hot spot''

Note that the ``Update'' button in the top button bar is again highlighted, as shown previously in Figure 2-6, indicating that new information is available in the trace file. Click on this button. Figure 2-13 shows the new display.

Figure 2-13 The main.c source display after executing wordcount.3

The new red block indicates that we need a test case where the input file cannot be opened, e.g., if the file does not exist. Scrolling through the source window to see other nonwhite blocks indicates that we need a test case where the line, word, and character counting options are specified explicitly. The following test cases cover these situations:

 	prompt:> wordcount nosuchfile    (wordcount.4) 
 	prompt:> wordcount -wlc input1   (wordcount.5)

The former test case should produce an error message indicating that the file could not be found and the latter test case should produce the same output as wordcount.1, as shown earlier.

Click on the ``Update'' button. Figure 2-14 shows the updated display. The scroll bar indicates that all blocks in the file have now been covered. To view the by file summary again, click on the ``Summary'' button in the top button bar. Figure 2-15 shows the new summary. As you can see, the five tests you have run so far have achieved 100% block coverage over both files. They do not, however, constitute a complete set of tests. There may be errors not revealed by these tests that will be revealed when different combinations of statements are executed, or when they are executed in a different order. The remaining coverage measures are designed to help create tests that will reveal these errors.

Figure 2-14 The main.c source display after executing wordcount.5	Figure 2-15 The coverage summary by-file after executing wordcount.5

Click on the ``by-type'' button. This will show the coverages achieved so far using various coverage measures, as shown in Figure 2-16. The first two entries indicate that the five tests you ran have covered all 3 of 3 function entries and 51 of 51 blocks in all source files. The next three entries provide the coverage status for other coverage measures known as decision, c-use, and p-use (see Section 3.3, What Does ATAC Do? for an explanation of these measures). Note that none of these measures have reached a 100% coverage status yet. Let us now try to raise the decision coverage to 100%.

Figure 2-16 The coverage summary by-type after executing wordcount.5

A decision is a conditional branch from one block to another. As can be seen from the coverage summary in Figure 2-16, it is possible that a set of tests will cover all blocks in a program without covering some of the decisions. In this example, 30 of the total 35 decisions have been covered. In order to determine what additional test cases are needed to cover the remaining five decisions, click on the ``decision'' button in the third row. Figure 2-17 shows the resulting display.

Figure 2-17 The main.c decision display after executing wordcount.5

Alternatively, to switch to decision coverage, you may also click on the ``Options'' button in the top button bar and select the ``decision coverage'' entry from the resulting menu, as shown in Figure 2-18.

Figure 2-18 The Options menu

In a decision display, all conditional expressions in a file are highlighted. If an expression is highlighted in white, it means all branches originating at that expression have been covered. If, on the other hand, it is highlighted in a nonwhite color, it means there is at least one branch originating there that has not been covered yet. To find out which one, you may click on the highlighted expression. This will pop up a window showing a list of all branches originating there highlighted in colors that indicate their coverage status and current weights.

The scroll bar shows that there are several expressions highlighted in red. Use the bottom (or top) arrow in the scroll bar to scroll up (or down) the source window by a few lines so the highlighted expression controlling the do-while loop becomes visible in the source window. Then click on the highlighted expression to pop up the list of all branches originating there, as shown in Figure 2-19.

Figure 2-19 A decision ``hot spot'' in main.c with a window showing the list of all branches originating there

Note that, of the two possible outcomes of the highlighted conditional expression, true and false, the latter is highlighted in white indicating that the false branch of the loop expression has already been covered. The former, however, is highlighted in red implying that the true branch is yet to be covered. Note that the loop expression itself is highlighted in red indicating that at least one of the branches originating there remains to be covered. In general, the color of a conditional expression at any time is the same as the color of the ``heaviest'' branch originating there at that time.

To cover the true branch of the loop expression, you must invoke wordcount with more than one input file. Execute the following command to do this:

 	prompt:> wordcount input1 input2   (wordcount.6)

It should produce the following output:

		1	 4	 19	input1
		2	 8	 38	input2
		3	12	 57	total

Click on the highlighted ``Update'' button to read the coverage information from the above test case. Figure 2-20 shows the relevant part of the updated display.

Figure 2-20 A part of the main.c decision display after executing wordcount.6	Figure 2-21 A decision ``hot spot'' in main.c with its branch list after executing wordcount.6

Click on the ``dismiss'' entry at the bottom of the decision branch list (Figure 2-20). This will remove the branch list window from the display.⁴

The scroll bar now indicates that there are still three nonwhite conditional expressions towards the bottom of the file and one towards the top of the file. Click near the top of the scroll bar to bring the corresponding text in the source window. Then click on the highlighted switch expression to show the corresponding branch list,as shown in Figure 2-21. The branch list indicates that four of the five possible branches of the switch statement have already been covered. The remaining branch can be covered by invoking the wordcount program with a ``-?'' option. The following command accomplishes this⁵:

 	prompt:> wordcount -?   (wordcount.7)

This test case should print an appropriate usage message. Click on the ``Update'' button to confirm that it has, indeed, covered the desired branch of the switch statement. Then close the branch list window by clicking on the ``dismiss'' entry.

So far we have invoked wordcount with options that have caused it to print all three - line, word, and character - counts. We have never invoked it to print only one or two of these counts. Examining the remaining uncovered decisions reveals that we should invoke the program with only one of the three options, -l, -w, and -c, in order to cover these uncovered decisions. The following two commands achieve this:

 	prompt:> wordcount -l input1   (wordcount.8)
 	prompt:> wordcount -w input1   (wordcount.9)

They should produce appropriate line and word counts, respectively, for the file, input1. Click on the ``Update'' button and you will see that all decisions in the file have been covered. Now go back to the summary window to check the overall coverage status by clicking on the ``Summary'' button and selecting the ``by-type'' coverage option. Figure 2-22 shows the new coverage summary.

Figure 2-22 The coverage summary by-type after executing wordcount.9

All blocks and decisions have now been covered. The fourth row in the summary display, however, indicates that there are some c-uses that have not been exercised. A c-use, or a computational variable use, is a combination of an assignment to a variable and a subsequent use of that variable in a computation that is not part of a conditional expression (see Section 3.3, What Does ATAC Do?). Typically one attempts to achieve high c-use coverage only for code which must be tested very thoroughly. Let us now try to cover the remaining c-uses that have not yet been covered.

Click on the ``c_use'' button in the summary display. Figure 2-23 shows the resulting display. A c-use display for a file highlights all the definitions of, or the assignments to, the variables involved in all c-uses in the file. If a c-use assignment is highlighted in white, it means all c-uses originating at that assignment have been covered. If, on the other hand, it is highlighted in a nonwhite color it means that there is at least one c-use originating at this assignment that has not been covered yet. For example, the white highlighting of the assignment to the variable p in the loop initialization of the for loop in Figure 2-23 indicates that all c-uses involving this assignment have already been covered. The assignment of the variable doline in the first switch branch inside the for loop, on the other hand, is highlighted in red. This means there are one or more c-uses of this assignment that have not been covered yet. To see which ones, click on the assignment statement. Figure 2-24 shows the resulting display after scrolling down the file so the highlighted c-uses become visible in the source window.

Figure 2-23 The c-use definitions display for main.c after executing wordcount.9	Figure 2-24 The display showing the c-uses of the highlighted assignment to the variable doline in Figure 2-23

Note that the highlighting of the assignment is also retained for easy reference, although in a different color so as not to confuse it with the corresponding uses.

Of the three c-uses of the assignment to doline, two are highlighted in white indicating that they have been covered. To cover the remaining uncovered c-use, we must invoke the program asking it explicitly to count the number of lines when the input is supplied via the standard input. The following test case achieves this:

 	prompt:> wordcount -l < input1   (wordcount.10)

Updating the display with the ``Update'' button confirms that the uncovered c-use of doline has been covered.

Click on the ``Summary'' button and select the ``c_use'' entry to go back to the c-use definitions display. Execute the following two tests to cover the analogous c-uses involving the assignments to the variables doword and dochar inside the switch statement in
Figure 2-23:

 	prompt:> wordcount -w < input1   (wordcount.11)
 	prompt:> wordcount -c < input1   (wordcount.12)

Examining the remaining uncovered c-uses reveals that we have never tested the program to see if it works correctly in the following situations:

Invoking the program with a valid command line option in combination with an input file that does not exist;

Invoking it with one valid and one invalid command line option at the same time;

Invoking it with one valid and one invalid input file at the same time.

Execute the following test cases to address the above situations:

 	prompt:> wordcount -l nosuchfile      (wordcount.13)
 	prompt:> wordcount -lx input1         (wordcount.14)
 	prompt:> wordcount input1 nosuchfile  (wordcount.15)

Update the display with the ``Update'' button. All c-uses in the file have now been covered. Display the overall coverage summary by clicking on the ``Summary'' button and selecting the ``by-type'' coverage option, as shown in Figure 2-25. The c-use summary indicates that only 89 of the total 92 c-uses have been covered yet. But all c-use assignments in main.c were highlighted in white indicating that all c-uses in that file have been covered. This means the remaining three uncovered c-uses must be in the file wc.c.

Figure 2-25 The coverage summary by-type after executing wordcount.15

To switch to wc.c display, click on the ``File'' button in the top button bar and select the ``wc.c'' entry from the resulting file menu. Figure 2-26 shows the c-use display for wc.c.

Figure 2-26 The c-use definitions display for wc.c after executing wordcount.15

Click on the first highlighted assignment. Figure 2-27 shows the corresponding c-uses. The only remaining uncovered c-use in the display can be covered if the control never enters the body of the while statement. This is possible only when the program is invoked with an empty input file. Execute the following test case to achieve this:

 	prompt:> wordcount empty   (wordcount.16)

Figure 2-27 The display showing the c-uses of the first highlighted assignment in Figure 2-26

Update the display using the ``Update'' button to check that the uncovered c-use is covered by the last test case. Click on the ``Summary'' button and select the ``c-use'' entry to go back to the c-use definitions display, as shown in Figure 2-28. Note that besides covering the desired c-use involving the variable nl, the last test case also covered the analogous c-uses involving the variables nw and nc. Figure 2-29 shows the new coverage summary. As the c-use entry in the summary indicates, you have now achieved a 100% c-use coverage.

Figure 2-28 The c-use definitions display for wc.c after executing wordcount.16	Figure 2-29 The coverage summary by-type after executing wordcount.16

The fifth row of the coverage summary in Figure 2-29 indicates the current p-use coverage status. A p-use, or a predicate variable use, is a combination of an assignment to a variable, a subsequent use of that variable in a conditional expression, and a particular branch originating at that conditional expression (see Section 3.3, What Does ATAC Do?). Thus a p-use is like a c-use except that the variable use is in a branch originating at a conditional expression. Note that 30 of the total 31 p-uses have already been covered.

To see the only remaining uncovered p-use, click on the ``p_use'' button in the summary display. Figure 2-30 shows the resulting display. Like the c-use display, the p-use display highlights all the definitions of, or assignments to, the variables involved in all p-uses in the file. If a p-use assignment is highlighted in white, it means all p-uses originating at that assignment have been covered. Four of the five highlighted assignments⁶ in Figure 2-30 are highlighted in white. The nonwhite color of the remaining assignment indicates that there is at least one p-use originating at that assignment that is yet to be covered. To see all the p-uses of this assignment, click on the highlighted assignment expression. Figure 2-31 shows the resulting display. It highlights all conditional expressions that use the variable assigned by the assignment in question. In this case there is only one such conditional expression, highlighted in red. Note that the assignment in question has also been highlighted in a different color for your reference. Recall that a p-use involves an assignment, a conditional expression, and a particular branch originating at the conditional expression. So far, we have only seen the former two elements of the remaining uncovered p-use. To see the last element, click on the highlighted conditional expression. A window containing a list of branches originating there pops up, as shown in Figure 2-32.

Figure 2-30 The p-use definitions display for wc.c after executing wordcount.16	Figure 2-31 The conditional expressions involved in p-uses originating at the red assignment in Figure 2-30
Figure 2-32 A part of the wc.c p-use display after executing wordcount.16	Figure 2-33 A part of the wc.c p-use display after executing wordcount.17

The color of a branch indicates the coverage status of the corresponding p-use. The red highlighting of the true branch indicates that the p-use involving the definition of the variable state outside the while loop and its subsequent use in the true branch of the highlighted expression inside the loop is yet to be covered. Analysis of the code reveals that this p-use will be covered only if the very first character read by the program is a nonwhite character. Both input files we have used so far, input1 and input2, contained a tab in their first character positions. On the other hand, input3 does not have white space at the beginning of the file. So the following command should cover the above p-use:

 	prompt:> wordcount input3   (wordcount.17)

Click on the ``Update'' button. Figure 2-33 shows the updated branch list indicating that the p-use we were trying to cover was indeed covered.

Note that the false entry in the branch list is not highlighted at all (Figure 2-32 and Figure 2-33) either in white or in a nonwhite color. This is because the corresponding p-use is an infeasible p-use -- it is impossible to cover it by any test case. The assignment involved assigns the value, OUT, to the variable, state. The conditional expression involved checks to see if state has the value OUT. Whenever the value examined by the latter is that assigned by the former, the conditional expression will evaluate to true. Thus it is impossible to cover the corresponding p-use involving the false branch. ATAC automatically detects many infeasible decisions, c-uses and p-uses and ignores them. It cannot, however, detect all such decisions, c-uses or p-uses.⁷

Click on the ``dismiss'' entry to close the branch list. Then click on the ``Summary'' button in the top button bar and select the ``by-type'' entry. Figure 2-34 shows the resulting coverage summary. All coverage criteria measured by ATAC are now covered. From ATAC's point of view, these 17 tests are a completely adequate test of wordcount. Of course, all we have done is create a set of tests that will thoroughly test the program. You must check that the program actually passed the tests. This may be done while using ATAC or after recompiling the program with the standard compiler.

Figure 2-34 The coverage summary by-type after executing wordcount.17

There is no guarantee that a program which has passed a completely adequate set of tests has no errors.⁸ However, in addition to producing test sets that reveal errors, the use of ATAC and ATAC to achieve high coverage places the source code under intensive scrutiny which also tends to reveal errors. A complete test set combined with the effort to create such a test set is very effective at revealing errors. For large programs, it may require extensive testing to achieve 100% block, decision, c-use, and p-use coverage as we did for the wordcount program. In practice it may be necessary to settle for less than 100% coverage.

To quit ATAC click on the ``File'' button in the top button bar, then select ``exit''.

---

¹ A basic block, or simply, a block, is a code sequence that is always executed sequentially, i.e., it has no internal branching constructs. It is also described as any ``single-entry-single-exit'' region of code; see Section 3.3, What Does ATAC Do?.

² Although the red region, in this case, appears to consist of a single block, it is a sequence of two contiguous basic blocks, as the first one of the two is a function call. A function call, in general, may never return, e.g., if it invokes exit under certain conditions. Thus a function call breaks the ``single-entry-single-exit'' property of a basic block and results in the start of a new basic block at a statement immediately following the function call.

³ If there are more files than can fit in the summary window, a scroll bar appears to the left of the window. You may use it to scroll through the list of files.

⁴ A branch list window that pops up when you click on a highlighted conditional expressions in a source window ``sticks'' to the display at the point where the mouse was clicked. It does not scroll up or down with the source window. Therefore you should always close it by clicking on the ``dismiss'' entry before scrolling the source window. You may, however, invoke ATAC with the -nosticky option to make its behavior similar to that of a pulldown menu. In that case, a branch list window will remain popped up as long as you keep the mouse button pressed. It will be automatically closed when you release the button.

⁵ You may need quotes around the question mark depending on whether or not the command processor you are using interprets it as a wildcard character.

⁶ Highlighting of a type declaration of a function argument, as in the case of the first highlighted definition in Figure 2-30, refers to the implicit assignment of the formal argument with the actual argument in the corresponding function call.

⁷ No program can automatically detect all infeasible decisions, c-uses or p-uses as the general problem of determining if a decision, a c-use or a p-use is infeasible is an unsolvable problem.

⁸ In general, only when all possible inputs have been tested does passing the tests imply the program is error free. For most programs this is impossible.

---

[Top] [Prev] [Next] [Index] [TOC]

---