Automatic Generation and Execution of Specification-Based Test Cases in a Network Based Environment

Abstract

Automatic testing of software has always been of interest. The concept has been receiving more and more attention in the software development climate of today (e.g., agile software processes) where managers and testers are being asked to turn around their products under shorter schedules and with smaller resources. Automation is being looked at as a pathway to meet the product testing and delivery deadlines. There are many research methods and tools, as well as a number of commercial tools that automate test design, generation and execution. Some are specification-based, some are structure based, and some are combinations of both. However the current state of automation of software testing has problems. These include design and maintainability of automation (e.g., current methods may not track product changes in a cost-effective way, and test-case suites do not always evolve easily and consistently), generation of excessive number of test-cases, end-user friendliness of the tool interfaces, measurability of the test-suite effectiveness, and so on.

The goal of the work presented here is to study automation of efficient specification-based test-case design, generation and execution. The investigation focuses on the principle known as pair-wise testing. Empirical evidence exists that shows that the technique can produce, depending on the algorithm used, test suites which are not only efficient in fault detection, but also have a manageable size. Objectives of the work were to a) enhance and further asses the open-source pair-wise test case design tool, developed previously by K.C. Tai and his students, called PairTest, b) develop a prototype of an adaptable easy-to-use system for automatic execution of the PairTest test cases, something that the original release of PairTest does not do, in a setting here one tests network-based systems, and c) assess usability of pair-wise testing strategy and automation of testing in an industrial environment.

This work explicitly recognizes two stages of testing automation: i) automated test-case design and generation, and ii) automated construction of executable test-case from the generated suite. In assessing performance of PairTest, it was compared to the automatic test case generation practices in the industry with the help of different user scenarios. The prototype system for conversion of PairTest generated test-cases into executable test-suites is based on the open-source 'Expect' toolset. PairTest performs favorably where it comes to test-case design and generation, as well as fault-detection power. It exhibits an ability to contain exponential growth in the number of tests generated with the increase in the number of specification parameter values. A quantitative analysis of pair-wise testing and test automation in two industrial environment shows that, compared with manual methods, automatic test design and execution in general help a) increase the error detection capability (efficiency) of testing, b) reduce resource consumption, and c) increase evolvability (maintainability, adaptability) of the testing process. Results confirm that properly automated testing, as a 'complete' testing solution, not only improves the quality of the software and system, but also saves time and resources.