Chapter 8

ATAC: Generating Summary Reports

When a program is compiled with atac, a .atac file is generated for each instrumented source file, and coverage information is appended to a trace file each time the compiled executable is tested. Each .atac file contains a list of what should be covered when testing its corresponding source file. The trace file records what has actually been covered during testing. ATAC uses the same facilities as does ATAC in reporting testing results, therefore the presentation of summary reports for ATAC and ATAC are merged in this chapter. This chapter discusses how to generate reports using these files that summarize the current level of code coverage and what each test case has contributed to this coverage. Such reports provide a high-level view of how a test is progressing and the relative contribution of each test case to the software testing effort.

The character-based interface, atac, accepts various command-line options, one or more .atac files, and a trace file as input. atac compares what should be covered in a given source file to what has been covered and outputs its findings in a variety of ways. The -s command- line option is used to generate summary reports. Additional arguments are used along with -s to further specify the contents and format of a report. For a synopsis of atac usage, see its command reference pages in Appendix B.2.

In the graphical interface, ATAC, summary report selections are made by clicking the ``Summary'' button in the top button bar and selecting the appropriate coverage option. The .atac and .trace files may be specified either on the command line or added later during a ATAC session using the ``File'' menu.

8.1 Generating Coverage Summaries

To invoke atac in summary generation mode, use the -s command-line option. For example, assume that the wordcount program discussed in Chapter 2, has been executed only against the following three tests:

	prompt:> wordcount -l empty
	prompt:> wordcount -w empty
	prompt:> wordcount -c empty

Then a summary of the current level of code coverage is obtained by entering:

	prompt:> atac -s main.atac wc.atac wordcount.trace

The summary report generated looks like this:



        % blocks      % decisions   % C-Uses      % P-Uses      
        ------------- ------------- ------------- ------------- 
        67(34/51)     49(17/35)     51(47/92)     42(13/31)     == total ==

This report indicates that, throughout all source files making up the wordcount program, 67% of all blocks (34 of 51), 49% of all decisions (17 of 35), 51% of all c-uses (47 of 92), and 42% of all p-uses (13 of 31) have been covered.

The corresponding summary in the graphical interface is obtained by clicking on the ``Summary'' button in the top button bar and selecting the ``by-type'' coverage button. Figure 8-1 shows the resulting display. In any discussion involving ATAC through out this chapter, we will assume that the three test cases mentioned above have been executed and ATAC has been invoked using the following command:



Figure 8-1 Coverage summary by-type

	prompt:> xatac main.atac wc.atac wordcount.trace

Note that ATAC displays the percentages as a bar chart. Each bar is actually made up of two bars, one contained inside the other. The length of the outer bar represents the maximum possible (100%) coverage for the corresponding measure and that of the inner bar represents the actual coverage attained so far for that measure. As the actual coverage increases, the length of the inner bar increases accordingly.

Both ATAC and ATAC use standard rounding in reporting results except that 100% coverage and 0% coverage are treated as special cases. No result is ever rounded up to 100% or down to 0%. Also, whereas ATAC rounds percentages to whole numbers, ATAC rounds them to their first decimal places.

8.2 Selecting What to Summarize

As mentioned earlier, ATAC uses various command-line options to specify the summary format, and ATAC uses various menus for the same purpose. You can break down the summary information by file, by function, or by test case.

8.2.1 By File

In the character-based interface, a coverage summary broken down by source file is printed using the -g command-line option. Assuming that the test cases of Section 8.1 have been executed, a per source file coverage summary for the wordcount program is printed by entering:

	prompt:> atac -sg main.atac wc.atac wordcount.trace

The summary report generated looks like this:



        % blocks      % decisions   % C-Uses      % P-Uses      source file
        ------------- ------------- ------------- ------------- ------------- 
        76(29/38)     70(16/23)     53(39/74)     69(11/16)     main.c
        38(5/13)      8(1/12)       44(8/18)      13(2/15)      wc.c
        67(34/51)     49(17/35)     51(47/92)     42(13/31)     == total ==

This right most column indicates whether the coverage levels presented correspond to main.c, wc.c, or the union of all the source files presented. Note that the -g command-line option cannot be used along with the -f command-line option (see Section 8.2.2, By Function).

In the graphical interface, a per file summary is obtained by clicking the ``Summary'' button, then selecting the ``by-file'' coverage option. Figure 8-2 shows the per file summary after the three tests mentioned above have been executed. Files listed in the ``Summary'' window can be sorted in different ways. Clicking on ``Sort_by'' in the middle button bar and selecting the ``name'' entry will sort files by name in alpha-numeric order. The ``num_covered,'' ``percentage_covered,'' ``num_uncovered,'' and ``percentage_uncovered'' entries sort files by the number or percentage of testable attributes covered or uncovered, respectively, according to the currently selected coverage type. The entry ``num_total_units'' sorts files by the total number of testable attributes with respect to the currently selected coverage type. The ``default_order'' selection performs the sort according to the order the .atac files occurred on the command line; and the entry ``dont_sort'' preserves the current state of the sort allowing new information to be appended to the end.



Figure 8-2 Block coverage summary by-file

Clicking with the right mouse button on a file label that is currently selected deselects it, and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the ``Disable all .h files'' or ``Disable all source files'' entry will deselect all .h or source files, respectively, whereas the ``Enable all .h files'' or ``Enable all source files'' entry will re- select all .h or source files, respectively.

In the character-based interface the per-file summary includes information about all four coverage types; however, in the graphical interface, it includes information only about the currently selected coverage type. When ATAC is first invoked, block coverage is selected by default. To see the per-file coverage for another coverage type, e.g., decision coverage, click on the ``Options'' button in the top button bar and select the corresponding coverage entry in the options menu that pops up. Figure 8-3 shows the ``Options'' menu and Figure 8-4 shows the per-file decision coverage summary after selecting the ``decision coverage'' entry of the options menu.

Figure 8-3 The Options menu

: 8.2.2 By Function
A code coverage summary broken down by function is printed using the -f command-line option in the character-based interface. Assuming that the test cases of Section 8.1 have been executed, a per function coverage summary for the wordcount program is printed by entering:
prompt:> atac -sf main.atac wc.atac wordcount.trace

The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 100(8) 100(6) 100(4) 100(6) print 38(5/13) 8(1/12) 44(8/18) 13(2/15) count 67(34/51) 49(17/35) 51(47/92) 42(13/31) == total ==
This right most column indicates whether the coverage levels presented correspond to main, print, count, or the union of all functions presented. Note that the -f command-line option cannot be used along with the -g command-line option (see Section 8.2.1, By File).
In the graphical interface, the per function coverage summary is obtained by selecting the ``Summary'' button in the top button bar and then the ``by-function'' coverage button. Figure 8-5 shows an example. Functions listed in the ``Summary'' window can be sorted in different ways. Clicking on ``Sort_by'' in the middle button bar and selecting the ``name'' entry will sort functions by name in alpha- numeric order. The ``num_covered,'' ``percentage_covered,'' ``num_uncovered'' and ``percentage_uncovered'' entries sort functions by the number or percentage of testable attributes covered or uncovered, respectively, according to the currently selected coverage type. The entry ``num_total_units'' sorts functions by the total number of testable attributes with respect to the currently selected coverage type. The ``default_order'' selection performs the sort according to the order of .atac files on the command line; and the entry ``dont_sort'' preserves the current state of the sort, allowing new information to be appended to the end.

Figure 8-5 Decision coverage summary by-function

Clicking with the right mouse button on a function label that is currently selected deselects it, and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the ``Disable all functions'' entry will deselect all functions, whereas the ``Enable all functions'' entry will re-select all functions.
Function names in the ``Summary'' window can be displayed with or without the corresponding file name. Clicking with the left mouse button on ``file_name'' and selecting the ``include file name'' entry will have the file name included, whereas the ``exclude file name'' entry excludes the file name.
Unlike the character-based interface which includes all coverage types and all functions in all specified files in the per-function summary display, the graphical interface shows the coverage statistics only for the currently selected coverage type. As in the case of the per- file summary display, a different coverage type may be selected by choosing the corresponding entry in the ``Options'' menu.

8.2.3 By Test Case
In the character-based interface, the -p command-line option may be used along with the -s option to generate reports that summarize coverage on a per test case basis. Assuming that the test cases of Section 8.1 have been executed, a per test case coverage summary for the wordcount program is printed by entering:
prompt:> atac -sp main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.1 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.2 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.3 67(34/51) 49(17/35) 51(47/92) 42(13/31) == all ==
This right most column indicates whether the coverage levels presented correspond to wordcount.1, wordcount.2, wordcount.3, or the union of all test cases presented.
In the graphical interface, the per test case summary is obtained by clicking on the ``TestCases'' button on the top button bar. Figure 8-6 shows the per test case summary after executing the three tests mentioned in Section 8.1. As in the case of per file and per function display, the per test case display also shows coverages only for the currently selected coverage type. But unlike the per function display, the per test case display shows the coverage achieved by each test case over all source files, not just the currently selected file.

Figure 8-6 Decision coverage summary by test case

Clicking with the right mouse button on a test case label that is currently selected deselects it, and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the ``Disable all tests'' entry will deselect all tests, whereas the ``Enable all tests'' entry will re-select all tests. The entry ``Disable zero coverage tests'' will disable tests which do not contribute any coverage with respect to the currently selected coverage type.
Clicking on ``Sort_by'' in the middle button bar and selecting the ``name'' entry will sort tests by name in alpha-numeric order. The ``num_covered'' entry sorts tests by the number of testable attributes covered with respect to the currently selected coverage type. The entry ``default_order'' sorts tests according to the order in which they were executed, and the entry ``dont_sort'' preserves the current state of the sort, allowing new information to be appended to the end.

8.3 Restricting Summary Information
It is possible to restrict the information included in a summary by the character-based interface in many ways. Restrictions are made by controlling the .atac files submitted in the argument list and using various command-line options. The graphical interface does not offer any additional facilities for restricting the summary display besides those discussed in Section 8.2, Selecting What to Summarize. The options discussed within this section may be used together freely.

8.3.1 By File
The coverage summary generated by atac is limited to those .atac files submitted in the argument list. For example, the per function summary generated in Section 8.2.2 is limited to those functions in main.c by entering:
prompt:> atac -sf main.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 100(8) 100(6) 100(4) 100(6) print 76(29/38) 70(16/23) 53(39/74) 69(11/16) == total ==

8.3.2 By Function
An atac summary is limited to a selected set of functions using the -F command-line option. For example, the per function summary generated in Section 8.2.2 is limited to the functions main and count by entering:
prompt:> atac -sf -F main -F count main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 38(5/13) 8(1/12) 44(8/18) 13(2/15) count 60(26/43) 38(11/29) 49(43/88) 28(7/25) == total ==
Note that multiple functions may be specified by repeating the -F option, and functions may be named using wild card characters. These wild cards are the same as those used by the UNIX-like command processor in naming files (*, ?, [...]). In some cases, quote marks may be needed to prevent the command processor from expanding the test names as file names.

8.3.3 By Coverage Criteria
An atac summary is limited to a selected set of coverage measures using the -m {bcdepu} command-line option. The argument to -m selects one or more coverage measures: e selects function-entry coverage, b block coverage, d decision coverage, c c-use coverage, p p-use coverage, and u all-uses coverage (see Section 3.3, What Does ATAC Do?, for an explanation of these measures). For example, the per function summary generated in Figure 8-5 is limited to block and decision coverage by entering:
prompt:> atac -sf -mbd main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions function ------------- ------------- ------------- 70(21/30) 59(10/17) main 100(8) 100(6) print 38(5/13) 8(1/12) count 67(34/51) 49(17/35) == total ==

8.4 Additional Test Case Summaries
As mentioned in Section 8.2.3, in the character-based interface, the -p command-line option may be used along with the -s option to generate reports that summarize coverage on a per test case basis. The character-based interface also provides various other ways to display test case coverage summaries. If you are following the examples in the first three sections of this chapter, clear out the .trace file by removing it. Then execute the wordcount program against the following three test cases:
prompt:> wordcount -? prompt:> wordcount empty prompt:> wordcount < empty
Also assume ATAC is not running and that atactm is used to rename (see Section 7.3, Naming Test Cases):
prompt:> atactm -n wordcount.1 -r help_test wordcount.trace prompt:> atactm -n wordcount.2 -r empty_file_test wordcount.trace prompt:> atactm -n wordcount.3 -r stdin_test wordcount.trace
and assign a cost (see Section 7.6, Assigning Cost to Test Cases) to each of these tests:
prompt:> atactm -n help_test.1 -c 10 wordcount.trace prompt:> atactm -n empty_file_test.1 -c 20 wordcount.trace prompt:> atactm -n stdin_test.1 -c 20 wordcount.trace
A per test case code coverage summary is obtained by entering:
prompt:> atac -sp main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
Additional arguments are available to sort or include other test case information in the summary (see Section B.2, atac). These arguments may be used together freely.

8.4.1 Including Cumulative Coverage
Cumulative test case coverage is included in a summary using the -q command-line option. For example, the following generates a cumulative summary of the test cases presented above:
prompt:> atac -sq main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses test (cumulative) (cumulative) (cumulative) (cumulative) ------------- ------------- ------------- ------------- ------------- 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 63(32/51) 34(12/35) 42(39/92) 35(11/31) empty_file_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) stdin_test.1
With respect to block coverage, help_test.1 achieved 14 percent coverage, empty_file_test.1 adds an additional 49 percent coverage (total of 63%) and stdin_test.1 adds another 6 percent coverage (total of 69%). However, these results have been presented as a cumulative running total. The -q option implies a per test case summary, so the -p option need not be explicitly included.

8.4.2 Including Test Cost
The user-assigned cost of executing each test case is included in a summary using the -K command-line option. For example, the following generates a summary of the test cases presented above, including test cost:
prompt:> atac -sK main.atac wc.atac wordcount.trace
The summary report generated looks like this:

cost % blocks % decisions % C-Uses % P-Uses test ------ ------------- ------------- ------------- ------------- ------------- 10 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 20 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 20 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 50 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
The -K option implies a per test case summary, so the -p option need not be explicitly included.

8.4.3 Sorting by Coverage
Test cases are sorted in order of decreasing additional coverage using the -S command-line option. For example, to sort the test cases summarized above enter:
prompt:> atac -sS main.atac wc.atac wordcount.trace
The summary report generated looks like this:

% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
These test cases have been listed in order of increasing effectiveness across all coverage measures. The -S option implies a per test case summary, so the -p option need not be explicitly included.
The same output can also be obtained in the graphical interface by invoking Regress, clicking on the ``Minimize_in'' button in the middle button bar and selecting ``greedy_order'' (see Section 11.4, Test set minimization and prioritization via a graphical user interface).

8.4.4 Sorting by Cumulative Cost per Additional Coverage
Test cases are sorted in order of decreasing cost per additional code coverage using the -Q command-line option. All costs are presented as cumulative values. For example, to summarize the sorted cumulative cost per additional coverage for the test cases executed above enter:
prompt:> atac -sQ main.atac wc.atac wordcount.trace
The summary report generated looks like this:

cost % blocks % decisions % C-Uses % P-Uses test (cum) (cumulative) (cumulative) (cumulative) (cumulative) ------ ------------- ------------- ------------- ------------- ------------- 20 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 30 63(32/51) 34(12/35) 42(39/92) 35(11/31) help_test.1 50 69(35/51) 40(14/35) 46(42/92) 42(13/31) stdin_test.1
This output is similar to that obtained using the -S option, however test cost is also considered. The -Q option implies a per test case summary, so the -p option need not be explicitly included. Note that using the -Q option is equivalent to using the option sequence -qKS.
The same output can also be obtained in the graphical interface by invoking Regress, clicking on the ``Minimize_in'' button in the middle button bar and selecting ``optimal_order'' (see Section 11.4, Test set minimization and prioritization via a graphical user interface).

8.5 Ignoring What is Out-of-Date
By default, the character-based interface issues an error message if any of a program's .atac files have been significantly modified since a test case represented in its trace file was executed. Modifications which change only comments and white space in a program do not significantly affect the .atac files. Other changes may affect the structure of a program and these significantly affect the .atac files. ATAC determines that a test is out-of-date by comparing the check sums of source files in test cases and .atac files, as appropriate (see Section 7.7, Dealing with Source Code Modifications). The graphical interface also issues a warning whenever it detects any test cases to be out-of-date with any source files. It also marks the out-of-date test cases as an `o' in the test case display.
If a .atac file has been modified since the last test case was executed, you can force the generation of atac summary reports using the -i command-line option. This will cause atac to ignore out of date information although warnings may be issued. In general, the -i option should be used with caution. However, it can be used to avoid re-running tests against source code known to be unmodified prior to the last system build. The graphical interface, however, currently does not provide a way to force such a display.

[Top] [Prev] [Next] [Index] [TOC]

8.4 Additional Test Case Summaries