MOTIF: A tool for Mutation Testing with Fuzzing

TL;DR

MOTIF introduces a mutation testing tool for C that uses grey-box fuzzing to generate test cases, effectively detecting faults in mutants and outperforming symbolic execution in fault detection.

Contribution

It presents a novel mutation testing approach for C programs using fuzzing, overcoming symbolic execution limitations in safety-critical CPS software.

Findings

Detects up to 47% more faults than symbolic execution.

Effective in generating fault-detecting test cases for C code.

Improves mutation testing coverage in safety-critical systems.

Abstract

Mutation testing consists of generating test cases that detect faults injected into software (generating mutants) which its original test suite could not. By running such an augmented set of test cases, it may discover actual faults that may have gone unnoticed with the original test suite. It is thus a desired practice for embedded software running in safety-critical cyber-physical systems (CPS). Unfortunately, the state-of-the-art tool targeting C, a typical language for CPS software, relies on symbolic execution, whose limitations often prevent its application. MOTIF overcomes such limitations by leveraging grey-box fuzzing tools to generate unit test cases in C that detect injected faults in mutants. Indeed, fuzzing tools automatically generate inputs by exercising the compiled version of the software under test guided by coverage feedback, thus overcoming the limitations of symbolic execution. Our empirical assessment has shown that it detects more faults than symbolic execution (i.e., up to 47 percentage points), when the latter is applicable.