Chapter 10
ATAC: Testing Modified Code

In this chapter we explain how to test the modified code. The same wordcount program as used in previous chapters is used here. To copy these files, create a new directory, cd to it, and copy the contents of the directory in which the tutorial files are installed into the new directory. For illustration, we will use (1) three c files: main_old.c, main.c and wc.c, (2) three data files: input1, input2 and input3 and (3) the tests_regress script.

The remainder of this chapter is organized as follows: Section 10.1 describes how to view the coverage of modified code and Section 10.2 explains how to select existing regression tests to revalidate the modified code.

10.1 Coverage of Modified Code

When main_old.c is updated to main.c, there is one addition, one deletion and one change as shown in Figure 10-1. A tutorial for the Diff tool is presented in Section 16.2.

Figure 10-1 Differences between main_old.c and main.c

	prompt:> atac cc -c main.c

	prompt:> atacICC /w0 /Q /c main.c

	prompt:> atacCL /nologo /w /c main.c

Compile wc.c without ATAC according to your setup:

UNIX:

	prompt:> cc -c wc.c

Windows with IBM C compiler:

	prompt:> icc /w0 /Q /c wc.c

Windows with Microsoft C compiler:

	prompt:> cl /nologo /w /c wc.c

Generate the executable as appropriate for your computing environment:

UNIX:

	prompt:> atac cc -o wordcount main.o wc.o

Windows with IBM C compiler:

	prompt:> atacICC /w0 /Q wc.obj main.obj /Fewordcount.exe

Windows with Microsoft C compiler:

	prompt:> atacCL /nologo /w wc.obj main.obj /link /out:wordcount.exe

After the compilation, one .atac file (main.atac for main.c) and the executable wordcount(.exe) are created. Note one .atac file is created for each instrumented .c file, i.e., the .c files compiled with ATAC.

Run the following atacdiff command

	prompt:> atacdiff main.c main_old.c						

Invoke ATAC with respect to the modified code:

	prompt:> xsuds main.atac main.dif

	prompt:> wordcount -x input1

	prompt:> wordcount -wlc input1

Figure 10-2 The initial display of the main ATAC window to show the modifications between main.c and main_old.c

Figure 10-3 shows these two tests also give 100% coverage with respect to other metrics. In fact, the modification from main_old.c to main.c does not involve a modification of any decision, c-use or p-use.

Figure 10-3 The coverage summary by-type after two test executions

10.2 Modification-Based Regression Test Selection

The selection of regression tests is made by using the execution trace files of the old program. This is because if it is done on the new program, then the whole advantage of modification-based selection vanishes. Selecting regression tests which execute the deleted or changed code in the old program is easy, but it requires more work to identify tests that execute the code added to the new program. One way to do so is to select the tests that execute the line of code which is just before and after the corresponding position of the added code in the new release. For example, if code at lines 10 to 15 has been added in release i', atacdiff will select line 9 (the one just before) and line 16 (the one just after) in release i for coverage analysis. In general, such a selection guarantees that if a test case executes lines 9 and 16 in release i, it also executes lines 10 to 15 in release i' unless the execution in release i' can jump into or out of the added code without going through lines 9 or 16. In practice, this does not happen often.

Once we have regression tests selected based on their execution of the modified code, we can go a step further, as discussed in Chapter 11, to apply test set minimization and test case prioritization to determine which tests, among those necessary, should be reexecuted first, and which ones have lower priority or are to be omitted from reexecution. For the moment, let us emphasize modification-based regression test selection using main_old.c and main.c as examples.

Once again, since all the modification is in main, we only need to compile this file with atac. However, instead of using main.c as in Section 10.1, main_old.c is used here. Note that before we compile main_old.c, we need to delete object files generated from the previous compilation. This can be done by executing the clean command appropriate for the setup you are using. See Appendix A, Platform Specific Information to determine this.

Compile main_old.c with ATAC by using the appropriate command below:

UNIX:

	prompt:> atac cc -c main_old.c

Windows with IBM C compiler:

	prompt:> atacICC /w0 /Q /c main_old.c

Windows with Microsoft C compiler:

	prompt:> atacCL /nologo /w /c main_old.c

Compile wc.c without ATAC, as appropriate for your computing environment:

UNIX:

	prompt:> cc -c wc.c

Windows with IBM C compiler:

	prompt:> icc /w0 /Q /c wc.c

Windows with Microsoft C compiler:

	prompt:> cl /nologo /w /c wc.c

Generate the executable, per your setup:

UNIX:

	prompt:> atac cc -o wordcount main_old.o wc.o

Windows with IBM C compiler:

	prompt:> atacICC /w0 /Q wc.obj main_old.obj /Fewordcount.exe

Windows with Microsoft C compiler:

	prompt:> atacCL /nologo /w wc.obj main_old.obj /link /out:wordcount.exe

After the compilation, one .atac file (main_old.atac for main_old.c) and the executable wordcount(.exe) are created.

Run the following atacdiff command

	prompt:> atacdiff main_old.c main.c						

Suppose before main_old.c was updated to main.c, all the tests in Figure 11-1 had already been executed. This can be done by executing the script tests_regress. While the script is running, you will see some `error' messages because the tests are executing lines of code designed to handle error conditions.

To find all the regression tests that execute the modified code from main_old.c to main.c, run the following command:

	prompt:> atac -t main_old.atac main_old.dif wordcount.trace