[Top] [Prev] [Next] [Index] [TOC]

Chapter 7

ATAC: Managing Your Test Cases

When a program is compiled with ATAC, coverage information is appended to a trace file each time the resulting executable is tested. This information records the time of test case execution, what is covered, and the occurrence of any abnormal conditions (e.g., trace file corruption). This chapter discusses how to manage your test cases by manipulating the contents of the trace file.

The contents of an ATAC trace file are not in human-readable form. However, the ATAC trace manager, atactm, can be used to manipulate its contents. Atactm can list, rename, extract, delete, and assign cost to test cases. Each of these operations can be qualified to apply to a selective subset of test cases within a trace file. Listing test cases is a display operation. Renaming, deleting, or assigning cost to tests can either be performed in-place, effecting the existing trace file, or the results can be directed to a new trace file. Extracting one or more test cases means to select and copy them to a new trace file. For a synopsis of atactm usage, see its manual page entry in Appendix B.5, atactm. It should be noted that some, but not all of the functionalities of atactm are available in ATAC.

7.1 Listing Test Cases

A list of all the tests represented in a given trace file is obtained using the -l or -L command- line options. For example, assuming that four test cases have been executed against the wordcount program, invoking atactm on wordcount.trace with the -l option: 

	prompt:> atactm -l wordcount.trace
yields an abbreviated list of, in this case, default test names (see Section 7.3, Naming Test Cases, for more details): 

	wordcount.1   wordcount.2   wordcount.3   wordcount.4
Alternatively, invoking atactm on wordcount.trace with the -L option: 

	prompt:> atactm -L wordcount.trace
yields a longer, more informative, list of test case information: 
	12/25/97-09:53:45  0:00:01 4.0  100 ---------- wordcount.1
	12/25/97-09:53:50  0:00:01 4.0  100 ---------- wordcount.2
	12/25/97-09:53:57  0:00:00 4.0  100 ---------- wordcount.3
	12/25/97-09:54:02  0:00:00 4.0  100 ---------- wordcount.4
Using -l can be convenient in conjunction with shell scripts to aid in performing actions on large numbers of test cases. Using -L provides more information. The test list contains six or seven columns for each test providing the execution date-time and duration of the test, the version of ATAC used, a user-assigned test cost (the default is 100), test trace attributes (---------- for none), the test name, and the uncompressed trace file if applicable.The significance of test cost is discussed in the Section 7.6, Assigning Cost to Test Cases.

The possible trace attribute values are:

Like the other atactm operations, test cases can be selectively listed by using the -n and -x command-line options along with the -l or -L options.The use of these options in selecting specific test cases is discussed in Section 7.2, Selecting Test Cases.

7.2 Selecting Test Cases

Atactm operates on the information within a trace file corresponding to one or more test cases. The supported operations (e.g., list, rename, delete, etc.) can be performed on all tests, or selectively, on a subset of the existing tests. This is achieved using the -n option to select the tests on which the operation is to be applied. For example, assume the same four tests as listed in Section 7.1, Listing Test Cases, have been executed against the wordcount program. The first two of these tests are listed by entering: 

	prompt:> atactm -l -n wordcount.1 -n wordcount.2 wordcount.trace
which results in the following output: 

	wordcount.1   wordcount.2
Notice that multiple -n arguments may be submitted on a single command-line. In such a case, the union of all the tests named are selected. Tests can also be named using wild card characters. These wild cards are the same as those used by the UNIX-like command processor in naming files (*, ?, [...]). So, entering the appropriate command below: 
	prompt:> atactm -l -n 'word*.[1-2]' wordcount.trace (UNIX)
or
	prompt:> atactm -l -n word*.[1-2] wordcount.trace (Windows)

results in the same listing as the previous example. In some cases, quote marks may be needed to prevent the shell from expanding the test names as file names.

Sometimes it is necessary to select all tests other than those that have been named. The -x option is used to select the compliment of all tests specified using one or more instances of the -n option. For example, entering: 

	prompt:> atactm -l -x -n 'word*.[1-2]' wordcount.trace (UNIX)
or
	prompt:> atactm -l -x -n word*.[1-2] wordcount.trace (Windows)

lists the third and fourth test cases, which were excluded from the output of the previous examples: 

	wordcount.3   wordcount.4

7.3 Naming Test Cases

By default, tests are named prog.n, where prog is the base name of the trace file and n is a number starting at 1. Unique numeric values are appended to all test names so that two tests never have the same name in a given trace file. See Section 7.1, Listing Test Cases, for an example listing of default test names.

A test can also be named explicitly by setting the ATAC_TEST environment variable to the test name prior to execution (see Section 4.1.5, ATAC_TEST). Valid test names consist of letters, digits, and underscores. As with default test names, a numeric suffix is added to the name automatically to make it unique (a numeric suffix should not be specified in the value of ATAC_TEST). For example, testing wordcount by entering: 

	prompt:> ATAC_TEST=empty wordcount empty (UNIX)
or
	prompt:> set ATAC_TEST=empty 
	prompt:> wordcount empty       (Windows)

results in the creation of a new test case named empty.1, as shown by the -l listing: 

	wordcount.1  wordcount.2  wordcount.3  wordcount.4  empty.1

7.3.1 Renaming Test Cases

An existing test can be renamed using the -r option of atactm. For example, assume the wordcount tests wordcount.1 through wordcount.4 are respectively designed to test each of the legal command-line options for wordcount. These options are -w, -c, -l, and -?. A more meaningful set of test names can be provided in wordcount.trace by entering the following series of commands: 
        prompt:> atactm -r w_option -n wordcount.1 wordcount.trace
	prompt:> atactm -r c_option -n wordcount.2 wordcount.trace
	prompt:> atactm -r l_option -n wordcount.3 wordcount.trace
	prompt:> atactm -r q_option -n wordcount.4 wordcount.trace
Now listing the test cases represented in wordcount.trace results in: 

	w_option.1   c_option   l_option.1   q_option.1   empty.1
The -r option specifies the new test name, while the -n option selects the old test to be renamed. A group of tests may also be renamed using wild cards, multiple instances of the -n option, or the -x option to name the old tests (see Section 7.2, Selecting Test Cases, for more details). For example, to rename the tests from the last example so their names are all related, enter: 
	prompt:> atactm -r option_test -n '*_option.*' wordcount.trace (UNIX)
or
	prompt:> atactm -r option_test -n *_option.* wordcount.trace (Windows)

Listing the test cases represented in wordcount.trace now results in: 

	option_test.1  option_test.3  empty.1
	option_test.2  option_test.4
Note that, to guarantee uniqueness, a different .n suffix has been appended to the new test name for each test case. Make sure that only the desired test names match when using wild cards, otherwise multiple test cases can be unintentionally renamed.

7.3.2 What's in a Name?

ATAC supports the notion of a test case as a collection of trace information associated with a single program execution. Furthermore, ATAC provides a set of operations that can be selectively applied to each test case's trace information. However, ATAC has no concept of how a specific test case is intended to test a program, nor does ATAC manage test case setup, inputs, or outputs.

Test names are what link ATAC trace information with the test plan for a program. It is important to name test cases so that it is possible to identify what they are designed to test, what setup is required to make them run, where their inputs and outputs are located, and so on. While executing tests, the tester should keep track of all the required information and map it to the appropriate ATAC test names. Descriptive test case names may facilitate this process. Doing so will help to make tests repeatable and facilitate test set minimization (see Section 11.2, Test set minimization via a character-based user interface), both of which aid in regression testing.

7.4 Extracting Test Cases and Merging Trace Files

Test cases can be selectively extracted from an existing trace file and copied to a new trace file using the -e command-line option. Extracting test cases copies data to a new trace file without deleting those test cases from the original trace file. For example, trace information for the test case named wordcount.1 is copied from wordcount.trace and placed in new.trace by entering: 

	prompt:> atactm -e -n option_test.1 -o new.trace wordcount.trace
The -n and -o options are required in order to select the test case(s) of interest and specify their destination. In addition, the -x argument may optionally be used (see Section 7.2, Selecting Test Cases). Note that, if new.trace already exists, it is overwritten and will now only contain test case wordcount.1.

Atactm may be used to merge trace files. To merge new.trace and wordcount.trace into merge.trace enter: 

	prompt:> atactm -o merge.trace wordcount.trace new.trace
In this case, merge.trace now contains two copies of the first test case: the copy from wordcount.trace, which is named option_test.1 and the copy from new.trace which has been renamed option_test.5. Note that tests with the same name are renumbered to insure that two test cases do not have the same name. If the -o option is omitted, the first trace file is overwritten by the merged trace file.

7.5 Deleting Test Cases

From time-to-time, it may become necessary to delete trace information corresponding to specific test executions. This may occur if the program under test is modified (see Section 7.7, Dealing with Source Code Modifications).

Deleting an entire trace file discards the trace information for all tests whose executions are recorded in that file. Alternatively, trace information can be selectively discarded by invoking atactm with the -d command-line option. For example, trace information for the test case named wordcount.1 is deleted from wordcount.trace by entering: 

	prompt:> atactm -d -n wordcount.1 wordcount.trace
The -n option is required to select the test case(s) to be deleted, and the -x argument may optionally be used (see Section 7.2, Selecting Test Cases). Alternatively, the result after deleting test wordcount.1 from wordcount.trace is placed in new.trace by entering: 

	prompt:> atactm -d -n wordcount.1 -o new.trace wordcount.trace
When the -o option is used, the trace file wordcount.trace remains unchanged. If new.trace already exists, it is overwritten and will now contain all test cases in wordcount.trace, except for wordcount.1.

7.6 Assigning Cost to Test Cases

ATAC associates a cost with each test case. By default, all tests are given an initial cost of 100. However, invoking atactm with the -c command-line option allows the cost of each test to be selectively reassigned any non-negative value. This value is determined by the user and should reflect the relative difficulty and expense of executing the test case. Depending on the nature of the program under test, this may be a function of execution time, tester time, test setup time, or other factors. For example, assume that a large file called /etc/termcap is submitted to the wordcount program and this test is named termcap. Furthermore, assume we regard this test to be twice as expensive to run as the default test case. We can reassign the cost of termcap as 200 by entering: 

	prompt:> atactm -c 200 -n termcap wordcount.trace
The -n option is required in order to select the test case(s) of interest. In addition, the -x argument may optionally be used for selection (see Section 7.2, Selecting Test Cases), while the -o argument may optionally be used to place the results in a new trace file.

The cost assignments of one or more test cases can be selectively displayed by invoking atactm using the -L command-line option (see Section 7.1, Listing Test Cases). ATAC uses these cost assignments to minimize the size of a test set based upon the relative cost and coverage characteristics of individual test cases (see Section 11.2, Test set minimization via a character-based user interface).

7.7 Dealing with Source Code Modifications

If a source file is modified while testing is underway, ATAC's list of what needs to be covered within the file becomes out-of-date (see Section 3.4, How Does ATAC Work?). This source file must be recompiled and the program under test relinked using the ATAC compiler. Furthermore, test cases must be rerun because they may be out-of-date in relation to the new executable program.

It may not be convenient or cost-effective to repeatedly recompile after each source code modification. If recompilation is deferred pending additional changes, it is suggested that you make the modifications to source file copies, rather than the actual files originally compiled with ATAC. If the originals are modified, ATAC may no longer be able to display highlighted code fragments correctly. Warning messages signaling this are issued if an attempt is made to display code fragments within a file modified more recently than any of its trace information. However, the highlighted code fragments are still displayed in the event that minor code changes will not misalign the display too much. This permits coverage testing and display activities to continue prior to recompilation and in the face of minor source code changes, without copying files or rerunning previous test cases.

When you rebuild the program under test you should delete the old trace file or the affected tests. If you forget to do this and begin to execute tests, you should either delete all test cases run prior to the rebuild (see Section 7.5, Deleting Test Cases) or extract all test cases run since the rebuild and overwrite the trace file (see Section 7.4, Extracting Test Cases and Merging Trace Files). Otherwise, ATAC tools may output error messages identifying obsolete test cases. For example, after wordcount source file main.c is substantially modified and wordcount rebuilt, execution of the command: 

	prompt:> atac -s -n option_test.1 main.atac wordcount.trace
would result in the following message being printed and no output produced: 

	error: main.c differs substantially in test option_test.1 		
            (12/24/97-15:25:39) from main.atac (12/25/97-13:04:38)
If the change appears to be inconsequential, the error is reduced to a warning and output is produced. If the change only affects spaces and comments, no error or warning is issued.

7.8 Concerning Trace File Compression

Trace file compression discards all redundant trace information within a trace file. If two distinct test cases overlap in terms of coverage, all trace information that redundantly records this coverage for each test case is consolidated and shared between the tests. This tends to reduce the size of the trace file. Moreover, if trace file compression is performed after each test execution, other than a small entry recording that a test has been run, the trace file only grows if a test case actually improves coverage.

Each time atactm is invoked, if any options other than -l and -L are specified, including no options, the trace file is automatically compressed. For example, if atactm is invoked on wordcount.trace as follows, compression will occur: 

	prompt:> atactm wordcount.trace
This feature is useful, under certain conditions, as a means of forcing trace file compression (see Section 6.2, Trace File Compression).


[Top] [Prev] [Next] [Index] [TOC]