Causes and Effects of Fitness Landscapes in System Test Generation: A Replication Study

TL;DR

This study investigates how fitness landscape properties influence search-based system test generation, providing insights that can guide the development of more effective testing algorithms for industrial web services.

Contribution

It replicates prior fitness landscape analysis from unit testing to system testing, using EvoMaster on 23 web services, offering new empirical insights into practical industrial testing.

Findings

Fitness landscape properties vary across different web services.

Insights can inform the design of better search algorithms and fitness functions.

Empirical analysis highlights challenges and opportunities in system test generation.

Abstract

Search-Based Software Testing (SBST) has seen several success stories in academia and industry. The effectiveness of a search algorithm at solving a software engineering problem strongly depends on how such algorithm can navigate the fitness landscape of the addressed problem. The fitness landscape depends on the used fitness function. Understanding the properties of a fitness landscape can help to provide insight on how a search algorithm behaves on it. Such insight can provide valuable information to researchers to being able to design novel, more effective search algorithms and fitness functions tailored for a specific problem. Due to its importance, few fitness landscape analyses have been carried out in the scientific literature of SBST. However, those have been focusing on the problem of unit test generation, e.g., with state-of-the-art tools such as EvoSuite. In this paper, we replicate one such existing study. However, in our work we focus on system test generation, with the state-of-the-art tool EvoMaster. Based on an empirical study involving the testing of 23 web services, this enables us to provide valuable insight into this important testing domain of practical industrial relevance.