When a program is compiled with atac, a .atac file is generated for each instrumented source file, and coverage information is appended to a trace file each time the compiled executable is tested. Each .atac file contains a list of what should be covered when testing its corresponding source file. The trace file records what has actually been covered during testing.
The character-based interface, atac, accepts various command-line options, one or more .atac files, and a trace file as input. atac compares what should be covered in a given source file to what has been covered and outputs its findings in a variety of ways. The -s command- line option is used to generate summary reports. Additional arguments are used along with -s to further specify the contents and format of a report. For a synopsis of atac usage, see its command reference pages in Appendix B.2.
In the graphical interface,
ATAC, summary
report selections are made by clicking the ``Summary'' button in the
top button bar and selecting the appropriate coverage option.
The .atac and .trace files may be specified either on the command line or added later
during a
ATAC session using the ``File'' menu.
8.1 Generating Coverage Summaries
To invoke atac in summary generation mode, use the -s command-line option. For
example, assume that the wordcount program discussed in Chapter 2, has been executed
only against the following three tests:
prompt:> wordcount -l empty prompt:> wordcount -w empty prompt:> wordcount -c emptyThen a summary of the current level of code coverage is obtained by entering:
prompt:> atac -s main.atac wc.atac wordcount.traceThe summary report generated looks like this:
This report indicates that, throughout all source files making up the wordcount program, 67% of all blocks (34 of 51), 49% of all decisions (17 of 35), 51% of all c-uses (47 of 92), and 42% of all p-uses (13 of 31) have been covered.
% blocks % decisions % C-Uses % P-Uses ------------- ------------- ------------- ------------- 67(34/51) 49(17/35) 51(47/92) 42(13/31) == total ==
The corresponding summary in the graphical interface is obtained by clicking on the ``Summary'' button in the top button bar and selecting the ``by-type'' coverage button. Figure 8-1 shows the resulting display. In any discussion involving
|
prompt:> xatac main.atac wc.atac wordcount.traceNote that
Both ATAC and
prompt:> atac -sg main.atac wc.atac wordcount.traceThe summary report generated looks like this:
This right most column indicates whether the coverage levels presented correspond to main.c, wc.c, or the union of all the source files presented. Note that the -g command-line option cannot be used along with the -f command-line option (see Section 8.2.2, By Function).
% blocks % decisions % C-Uses % P-Uses source file ------------- ------------- ------------- ------------- ------------- 76(29/38) 70(16/23) 53(39/74) 69(11/16) main.c 38(5/13) 8(1/12) 44(8/18) 13(2/15) wc.c 67(34/51) 49(17/35) 51(47/92) 42(13/31) == total ==
In the graphical interface, a per file summary is obtained by clicking the ``Summary'' button, then selecting the ``by-file'' coverage option. Figure 8-2 shows the per file summary after the three tests mentioned above have been executed. Files listed in the ``Summary'' window can be sorted in different ways. Clicking on ``Sort_by'' in the middle button bar and selecting the ``name'' entry will sort files by name in alpha-numeric order. The ``num_covered,'' ``percentage_covered,'' ``num_uncovered,'' and ``percentage_uncovered'' entries sort files by the number or percentage of testable attributes covered or uncovered, respectively, according to the currently selected coverage type. The entry ``num_total_units'' sorts files by the total number of testable attributes with respect to the currently selected coverage type. The ``default_order'' selection performs the sort according to the order the .atac files occurred on the command line; and the entry ``dont_sort'' preserves the current state of the sort allowing new information to be appended to the end.
|
Clicking with the right mouse button on a file label that is currently selected deselects it,
and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the
``Disable all .h files'' or ``Disable all source files'' entry will deselect all .h or source files,
respectively, whereas the ``Enable all .h files'' or ``Enable all source files'' entry will re-
select all .h or source files, respectively.
ATAC is first invoked, block coverage is selected
by default. To see the per-file coverage for another coverage type, e.g., decision coverage,
click on the ``Options'' button in the top button bar and select the corresponding coverage
entry in the options menu that pops up. Figure 8-3
shows the ``Options'' menu and
Figure 8-4
shows the per-file decision coverage summary after selecting the ``decision
coverage'' entry of the options menu.
|
|
8.2.2 By Function
A code coverage summary broken down by function is printed using the -f command-line
option in the character-based interface. Assuming that the test cases of Section 8.1 have
been executed, a per function coverage summary for the wordcount program is printed by
entering:
prompt:> atac -sf main.atac wc.atac wordcount.trace
The summary report generated looks like this:
% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 100(8) 100(6) 100(4) 100(6) print 38(5/13) 8(1/12) 44(8/18) 13(2/15) count 67(34/51) 49(17/35) 51(47/92) 42(13/31) == total ==This right most column indicates whether the coverage levels presented correspond to main, print, count, or the union of all functions presented. Note that the -f command-line option cannot be used along with the -g command-line option (see Section 8.2.1, By File).
In the graphical interface, the per function coverage summary is obtained by selecting the ``Summary'' button in the top button bar and then the ``by-function'' coverage button. Figure 8-5 shows an example. Functions listed in the ``Summary'' window can be sorted in different ways. Clicking on ``Sort_by'' in the middle button bar and selecting the ``name'' entry will sort functions by name in alpha- numeric order. The ``num_covered,'' ``percentage_covered,'' ``num_uncovered'' and ``percentage_uncovered'' entries sort functions by the number or percentage of testable attributes covered or uncovered, respectively, according to the currently selected coverage type. The entry ``num_total_units'' sorts functions by the total number of testable attributes with respect to the currently selected coverage type. The ``default_order'' selection performs the sort according to the order of .atac files on the command line; and the entry ``dont_sort'' preserves the current state of the sort, allowing new information to be appended to the end.
|
Clicking with the right mouse button on a function label that is currently selected deselects
it, and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the
``Disable all functions'' entry will deselect all functions, whereas the ``Enable all
functions'' entry will re-select all functions.
8.2.3 By Test Case
In the character-based interface, the -p command-line option may be used along with the -s option to generate reports that summarize coverage on a per test case basis. Assuming
that the test cases of Section 8.1 have been executed, a per test case coverage summary for
the wordcount program is printed by entering:
prompt:> atac -sp main.atac wc.atac wordcount.traceThe summary report generated looks like this:
This right most column indicates whether the coverage levels presented correspond to wordcount.1, wordcount.2, wordcount.3, or the union of all test cases presented.
% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.1 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.2 59(30/51) 34(12/35) 42(39/92) 32(10/31) wordcount.3 67(34/51) 49(17/35) 51(47/92) 42(13/31) == all ==
In the graphical interface, the per test case summary is obtained by clicking on the ``TestCases'' button on the top button bar. Figure 8-6 shows the per test case summary after executing the three tests mentioned in Section 8.1. As in the case of per file and per function display, the per test case display also shows coverages only for the currently selected coverage type. But unlike the per function display, the per test case display shows the coverage achieved by each test case over all source files, not just the currently selected file.
|
Clicking with the right mouse button on a test case label that is currently selected deselects
it, and vice versa. Clicking with the left mouse button on ``Disable'' and selecting the
``Disable all tests'' entry will deselect all tests, whereas the ``Enable all tests'' entry will
re-select all tests. The entry ``Disable zero coverage tests'' will disable tests which do not
contribute any coverage with respect to the currently selected coverage type.
8.3 Restricting Summary Information
It is possible to restrict the information included in a summary by the character-based
interface in many ways. Restrictions are made by controlling the .atac files submitted in the
argument list and using various command-line options. The graphical interface does not
offer any additional facilities for restricting the summary display besides those discussed
in Section 8.2, Selecting What to Summarize. The options discussed within this section may
be used together freely.
8.3.1 By File
The coverage summary generated by atac is limited to those .atac files submitted in the
argument list. For example, the per function summary generated in Section 8.2.2 is limited
to those functions in main.c by entering:
prompt:> atac -sf main.atac wordcount.traceThe summary report generated looks like this:
% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 100(8) 100(6) 100(4) 100(6) print 76(29/38) 70(16/23) 53(39/74) 69(11/16) == total ==
prompt:> atac -sf -F main -F count main.atac wc.atac wordcount.traceThe summary report generated looks like this:
Note that multiple functions may be specified by repeating the -F option, and functions may be named using wild card characters. These wild cards are the same as those used by the UNIX-like command processor in naming files (*, ?, [...]). In some cases, quote marks may be needed to prevent the command processor from expanding the test names as file names.
% blocks % decisions % C-Uses % P-Uses function ------------- ------------- ------------- ------------- ------------- 70(21/30) 59(10/17) 50(35/70) 50(5/10) main 38(5/13) 8(1/12) 44(8/18) 13(2/15) count 60(26/43) 38(11/29) 49(43/88) 28(7/25) == total ==
prompt:> atac -sf -mbd main.atac wc.atac wordcount.traceThe summary report generated looks like this:
% blocks % decisions function ------------- ------------- ------------- 70(21/30) 59(10/17) main 100(8) 100(6) print 38(5/13) 8(1/12) count 67(34/51) 49(17/35) == total ==
prompt:> wordcount -? prompt:> wordcount empty prompt:> wordcount < empty
Also assume ATAC is not running and that atactm is used to rename (see Section 7.3,
Naming Test Cases):
prompt:> atactm -n wordcount.1 -r help_test wordcount.trace prompt:> atactm -n wordcount.2 -r empty_file_test wordcount.trace prompt:> atactm -n wordcount.3 -r stdin_test wordcount.trace
and assign a cost (see Section 7.6, Assigning Cost to Test Cases) to each of these tests:
prompt:> atactm -n help_test.1 -c 10 wordcount.trace prompt:> atactm -n empty_file_test.1 -c 20 wordcount.trace prompt:> atactm -n stdin_test.1 -c 20 wordcount.trace
A per test case code coverage summary is obtained by entering:
prompt:> atac -sp main.atac wc.atac wordcount.traceThe summary report generated looks like this:
Additional arguments are available to sort or include other test case information in the summary (see Section B.2, atac). These arguments may be used together freely.
% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
8.4.1 Including Cumulative Coverage
Cumulative test case coverage is included in a summary using the -q command-line option.
For example, the following generates a cumulative summary of the test cases presented
above:
prompt:> atac -sq main.atac wc.atac wordcount.traceThe summary report generated looks like this:
With respect to block coverage, help_test.1 achieved 14 percent coverage, empty_file_test.1 adds an additional 49 percent coverage (total of 63%) and stdin_test.1 adds another 6 percent coverage (total of 69%). However, these results have been presented as a cumulative running total. The -q option implies a per test case summary, so the -p option need not be explicitly included.
% blocks % decisions % C-Uses % P-Uses test (cumulative) (cumulative) (cumulative) (cumulative) ------------- ------------- ------------- ------------- ------------- 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 63(32/51) 34(12/35) 42(39/92) 35(11/31) empty_file_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) stdin_test.1
prompt:> atac -sK main.atac wc.atac wordcount.traceThe summary report generated looks like this:
The -K option implies a per test case summary, so the -p option need not be explicitly included.
cost % blocks % decisions % C-Uses % P-Uses test ------ ------------- ------------- ------------- ------------- ------------- 10 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 20 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 20 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 50 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
prompt:> atac -sS main.atac wc.atac wordcount.traceThe summary report generated looks like this:
These test cases have been listed in order of increasing effectiveness across all coverage measures. The -S option implies a per test case summary, so the -p option need not be explicitly included.
% blocks % decisions % C-Uses % P-Uses test ------------- ------------- ------------- ------------- ------------- 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 14(7/51) 11(4/35) 7(6/92) 6(2/31) help_test.1 37(19/51) 17(6/35) 18(17/92) 23(7/31) stdin_test.1 69(35/51) 40(14/35) 46(42/92) 42(13/31) == all ==
The same output can also be obtained in the graphical interface by invoking
8.4.4 Sorting by Cumulative Cost per Additional Coverage
Test cases are sorted in order of decreasing cost per additional code coverage using the -Q
command-line option. All costs are presented as cumulative values. For example, to
summarize the sorted cumulative cost per additional coverage for the test cases executed
above enter:
prompt:> atac -sQ main.atac wc.atac wordcount.traceThe summary report generated looks like this:
This output is similar to that obtained using the -S option, however test cost is also considered. The -Q option implies a per test case summary, so the -p option need not be explicitly included. Note that using the -Q option is equivalent to using the option sequence -qKS.
cost % blocks % decisions % C-Uses % P-Uses test (cum) (cumulative) (cumulative) (cumulative) (cumulative) ------ ------------- ------------- ------------- ------------- ------------- 20 53(27/51) 26(9/35) 38(35/92) 32(10/31) empty_file_test.1 30 63(32/51) 34(12/35) 42(39/92) 35(11/31) help_test.1 50 69(35/51) 40(14/35) 46(42/92) 42(13/31) stdin_test.1
The same output can also be obtained in the graphical interface by invoking
8.5 Ignoring What is Out-of-Date
By default, the character-based interface issues an error message if any of a program's .atac
files have been significantly modified since a test case represented in its trace file was
executed. Modifications which change only comments and white space in a program do not
significantly affect the .atac files. Other changes may affect the structure of a program and
these significantly affect the .atac files. ATAC determines that a test is out-of-date by
comparing the check sums of source files in test cases and .atac files, as appropriate (see
Section 7.7, Dealing with Source Code Modifications). The graphical interface also issues
a warning whenever it detects any test cases to be out-of-date with any source files. It also
marks the out-of-date test cases as an `o' in the test case display.