SN4KE: Practical Mutation Testing at Binary Level

TL;DR

This paper introduces SN4KE, a practical binary mutation testing approach using binary rewriting, which enhances bug detection and scalability for binary-only software by employing diverse mutation operators and optimized rewriting techniques.

Contribution

It presents a novel binary mutation analysis method with a rich set of mutation operators and evaluates its effectiveness and scalability on real-world benchmarks.

Findings

Rich mutation operators increase mutant diversity.

Reassembleable disassembly offers better scalability.

Higher mutation scores improve test suite evaluation.

Abstract

Mutation analysis is an effective technique to evaluate a test suite adequacy in terms of revealing unforeseen bugs in software. Traditional source- or IR-level mutation analysis is not applicable to the software only available in binary format. This paper proposes a practical binary mutation analysis via binary rewriting, along with a rich set of mutation operators to represent more realistic bugs. We implemented our approach using two state-of-the-art binary rewriting tools and evaluated its effectiveness and scalability by applying them to SPEC CPU benchmarks. Our analysis revealed that the richer mutation operators contribute to generating more diverse mutants, which, compared to previous works leads to a higher mutation score for the test harness. We also conclude that the reassembleable disassembly rewriting yields better scalability in comparison to lifting to an intermediate representation and performing a full translation.