For a multiple release software product, testers may be more concerned with whether the code that has been changed, added or deleted from one release to the next has been properly tested rather than the overall coverage with respect to a module, a subsystem or the entire software. They need to either create new tests or select existing regression tests to validate the modified code in order to make sure the new software still behaves the same way as the previous version, except where changes are expected. This can be done by first running atacdiff to find the difference between two releases followed by a coverage analysis on the generated .dif files and other related .atac and .trace files.
In this chapter we explain how to test the modified code. The same wordcount program as used in previous chapters is used here. To copy these files, create a new directory, cd to it, and copy the contents of the directory in which the tutorial files are installed into the new directory. For illustration, we will use (1) three c files: main_old.c, main.c and wc.c, (2) three data files: input1, input2 and input3 and (3) the tests_regress script.
The remainder of this chapter is organized as follows: Section 10.1 describes how to view the coverage of modified code and Section 10.2 explains how to select existing regression tests to revalidate the modified code.
10.1 Coverage of Modified Code
To view coverage of modified code only, run atacdiff to create a .dif and open it whenever
the corresponding .atac is used.
Diff tool is presented
in Section 16.2.
|
prompt:> atac cc -c main.c
prompt:> atacICC /w0 /Q /c main.c
prompt:> atacCL /nologo /w /c main.c
Compile wc.c without ATAC according to your setup:
prompt:> cc -c wc.c
prompt:> icc /w0 /Q /c wc.c
prompt:> cl /nologo /w /c wc.c
Generate the executable as appropriate for your computing environment:
prompt:> atac cc -o wordcount main.o wc.o
prompt:> atacICC /w0 /Q wc.obj main.obj /Fewordcount.exe
prompt:> atacCL /nologo /w wc.obj main.obj /link /out:wordcount.exe
After the compilation, one .atac file (main.atac for main.c) and the executable
wordcount(.exe) are created. Note one .atac file is created for each instrumented .c file, i.e.,
the .c files compiled with ATAC.
prompt:> atacdiff main.c main_old.cto generate main.dif which contains the differences between main.c and main_old.c. See Appendix B.6, atacdiff for more details.
Invoke
ATAC with respect to the modified code:
prompt:> xsuds main.atac main.difScroll down the
prompt:> wordcount -x input1and
prompt:> wordcount -wlc input1
|
![]() |
Figure 10-3 shows these two tests also give 100% coverage with respect to other metrics. In fact, the modification from main_old.c to main.c does not involve a modification of any decision, c-use or p-use.
|
The selection of regression tests is made by using the execution trace files of the old program. This is because if it is done on the new program, then the whole advantage of modification-based selection vanishes. Selecting regression tests which execute the deleted or changed code in the old program is easy, but it requires more work to identify tests that execute the code added to the new program. One way to do so is to select the tests that execute the line of code which is just before and after the corresponding position of the added code in the new release. For example, if code at lines 10 to 15 has been added in release i', atacdiff will select line 9 (the one just before) and line 16 (the one just after) in release i for coverage analysis. In general, such a selection guarantees that if a test case executes lines 9 and 16 in release i, it also executes lines 10 to 15 in release i' unless the execution in release i' can jump into or out of the added code without going through lines 9 or 16. In practice, this does not happen often.
Once we have regression tests selected based on their execution of the modified code, we can go a step further, as discussed in Chapter 11, to apply test set minimization and test case prioritization to determine which tests, among those necessary, should be reexecuted first, and which ones have lower priority or are to be omitted from reexecution. For the moment, let us emphasize modification-based regression test selection using main_old.c and main.c as examples.
Once again, since all the modification is in main, we only need to compile this file with atac. However, instead of using main.c as in Section 10.1, main_old.c is used here. Note that before we compile main_old.c, we need to delete object files generated from the previous compilation. This can be done by executing the clean command appropriate for the setup you are using. See Appendix A, Platform Specific Information to determine this.
Compile main_old.c with ATAC by using the appropriate command below:
prompt:> atac cc -c main_old.c
prompt:> atacICC /w0 /Q /c main_old.c
prompt:> atacCL /nologo /w /c main_old.c
Compile wc.c without ATAC, as appropriate for your computing environment:
prompt:> cc -c wc.c
prompt:> icc /w0 /Q /c wc.c
prompt:> cl /nologo /w /c wc.c
prompt:> atac cc -o wordcount main_old.o wc.o
prompt:> atacICC /w0 /Q wc.obj main_old.obj /Fewordcount.exe
prompt:> atacCL /nologo /w wc.obj main_old.obj /link /out:wordcount.exe
After the compilation, one .atac file (main_old.atac for main_old.c) and the executable
wordcount(.exe) are created.
prompt:> atacdiff main_old.c main.cto generate main_old.dif which contains the differences between main_old.c and main.c.
Suppose before main_old.c was updated to main.c, all the tests in Figure 11-1 had already been executed. This can be done by executing the script tests_regress. While the script is running, you will see some `error' messages because the tests are executing lines of code designed to handle error conditions.
prompt:> atac -t main_old.atac main_old.dif wordcount.traceOf the 17 tests, only six are selected as shown in Figure 10-4. This implies a 64.7% savings in terms of the number of regression tests that need to be reexecuted for program revalidation.
|