DSpot: Test Amplification for Automatic Assessment of Computational Diversity

TL;DR

This paper introduces DSpot, a test amplification technique that automatically assesses computational diversity among program variants by exploring input and output spaces, revealing behavioral differences outside specified input domains.

Contribution

The work presents a novel method using test amplification to quantify computational diversity, enhancing detection of behavioral differences in program variants beyond traditional testing.

Findings

Test amplification increases input coverage tenfold.

Effective detection of software diversity among variants.

Behavioral diversity originates from flexible code areas.

Abstract

Context: Computational diversity, i.e., the presence of a set of programs that all perform compatible services but that exhibit behavioral differences under certain conditions, is essential for fault tolerance and security. Objective: We aim at proposing an approach for automatically assessing the presence of computational diversity. In this work, computationally diverse variants are defined as (i) sharing the same API, (ii) behaving the same according to an input-output based specification (a test-suite) and (iii) exhibiting observable differences when they run outside the specified input space. Method: Our technique relies on test amplification. We propose source code transformations on test cases to explore the input domain and systematically sense the observation domain. We quantify computational diversity as the dissimilarity between observations on inputs that are outside the specified domain. Results: We run our experiments on 472 variants of 7 classes from open-source, large and thoroughly tested Java classes. Our test amplification multiplies by ten the number of input points in the test suite and is effective at detecting software diversity. Conclusion: The key insights of this study are: the systematic exploration of the observable output space of a class provides new insights about its degree of encapsulation; the behavioral diversity that we observe originates from areas of the code that are characterized by their flexibility (caching, checking, formatting, etc.).