A Factorial Experiment on Scalability of Search Based Software Testing

TL;DR

This paper compares the efficiency of search-based algorithms, especially genetic algorithms, against random testing for software test-data generation using automatically generated complex SUTs.

Contribution

It introduces a factorial experiment with automatically generated SUTs of increasing complexity to evaluate search-based testing methods versus random testing.

Findings

Genetic algorithms outperform random testing on complex SUTs

Automated program generation allows scalable testing scenarios

Results guide automation choices in software testing

Abstract

Software testing is an expensive process, which is vital in the industry. Construction of the test-data in software testing requires the major cost and to decide which method to use in order to generate the test data is important. This paper discusses the efficiency of search-based algorithms (preferably genetic algorithm) versus random testing, in soft- ware test-data generation. This study differs from all previous studies due to sample programs (SUTs) which are used. Since we want to in- crease the complexity of SUTs gradually, and the program generation is automatic as well, Grammatical Evolution is used to guide the program generation. SUTs are generated according to the grammar we provide, with different levels of complexity. SUTs will first undergo genetic al- gorithm and then random testing. Based on the test results, this paper recommends one method to use for automation of software testing.