In Regression Testing without Code

Abstract

Software products are often built from commercial-off-the-shelf (COTS) components. When new releases of these components are made available for integration and testing, source code is usually not provided by the COTS vendors. Various regression test selection (RTS) techniques have been developed and have been shown to be cost effective. However, the majority of these test selection techniques rely on source code for change identification and impact analysis. This dissertation presents an RTS process called Integrated - Black-box Approach for Component Change Identification (I-BACCI) for COTS-based applications. The I-BACCI process reduces the test suite based upon changes in the binary code of the COTS component using the firewall analysis RTS method. This dissertation also presents Pallino, the supporting automation that statically analyzes binary code to identify the code change and the impact of these changes. Based on the output of Pallino and the original test suit, testers can determine the regression test cases needed that execute the application glue code which is affected by the changed areas in the new version of the COTS component.

Five case studies were conducted on ABB internal products written in C⁄C++ to determine the effectiveness and efficiency of the I-BACCI process. The total size of application and component for each release is about 340˜930 KLOC. The results indicate this process can reduce the required number of regression test by as much as 100% if there are a small number of changes in the new component in the best case. Similar to other RTS techniques, when there are many changes in the new component the I-BACCI process suggests a retest-all regression test strategy.

With the help of Pallino, RTS via the I-BACCI process can be completed in about one to two person hours for each release of the case studies. Depending upon the percentage of test cases reduction determined by the I-BACCI process, the total time cost of the whole regression testing process can be reduced to 0.0003% of that by retest-all strategy in the best case. Currently, Pallino works on components in Common Object File Format or Portable Executable formats.