Fuzzing for CPS Mutation Testing

TL;DR

This paper introduces a fuzz testing approach for mutation testing in safety-critical CPS software, demonstrating it kills more mutants than symbolic execution, especially when symbolic methods are infeasible.

Contribution

It presents the first empirical comparison of fuzz testing and symbolic execution for mutation testing in CPS, showing fuzz testing's effectiveness and guiding future tool development.

Findings

Fuzz testing kills up to 47% more mutants than symbolic execution.

Fuzz testing is especially beneficial when symbolic execution cannot be applied.

The approach improves mutation testing effectiveness for satellite system software.

Abstract

Mutation testing can help reduce the risks of releasing faulty software. For such reason, it is a desired practice for the development of embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, state-of-the-art test data generation techniques for mutation testing of C and C++ software, two typical languages for CPS software, rely on symbolic execution, whose limitations often prevent its application (e.g., it cannot test black-box components). We propose a mutation testing approach that leverages fuzz testing, which has proved effective with C and C++ software. Fuzz testing automatically generates diverse test inputs that exercise program branches in a varied number of ways and, therefore, exercise statements in different program states, thus maximizing the likelihood of killing mutants, our objective. We performed an empirical assessment of our approach with software components used in satellite systems currently in orbit. Our empirical evaluation shows that mutation testing based on fuzz testing kills a significantly higher proportion of live mutants than symbolic execution (i.e., up to an additional 47 percentage points). Further, when symbolic execution cannot be applied, fuzz testing provides significant benefits (i.e., up to 41% mutants killed). Our study is the first one comparing fuzz testing and symbolic execution for mutation testing; our results provide guidance towards the development of fuzz testing tools dedicated to mutation testing.