Search-based Software Testing Driven by Automatically Generated and Manually Defined Fitness Functions

TL;DR

This paper introduces ATheNA, a novel search-based software testing framework that combines automatically generated and manually defined fitness functions to improve failure detection in software models.

Contribution

ATheNA framework integrates automated and manual fitness functions, enhancing failure detection in SBST, with an implementation targeting Simulink models and comprehensive evaluation.

Findings

ATheNA-S outperforms baseline tools in revealing failures.

Runtime performance of ATheNA-S is comparable to baselines.

Successfully detects requirement violations in case studies.

Abstract

Search-based software testing (SBST) typically relies on fitness functions to guide the search exploration toward software failures. There are two main techniques to define fitness functions: (a) automated fitness function computation from the specification of the system requirements, and (b) manual fitness function design. Both techniques have advantages. The former uses information from the system requirements to guide the search toward portions of the input domain more likely to contain failures. The latter uses the engineers' domain knowledge. We propose ATheNA, a novel SBST framework that combines fitness functions automatically generated from requirements specifications and those manually defined by engineers. We design and implement ATheNA-S, an instance of ATheNA that targets Simulink models. We evaluate ATheNA-S by considering a large set of models from different domains. Our results show that ATheNA-S generates more failure-revealing test cases than existing baseline tools and that the difference between the runtime performance of ATheNA-S and the baseline tools is not statistically significant. We also assess whether ATheNA-S could generate failure-revealing test cases when applied to two representative case studies: one from the automotive domain and one from the medical domain. Our results show that ATheNA-S successfully revealed a requirement violation in our case studies.