Substate Profiling for Effective Test Suite Reduction

TL;DR

This paper introduces Substate Profiling, a novel, fine-grained state profiling technique that improves test suite reduction and defect detection by capturing program variable states more effectively than existing structural profiles.

Contribution

The paper presents Substate Profiling, a new state profiling method that enhances test suite reduction and fault localization, outperforming traditional structural profiles in defect detection.

Findings

Substate Profiling detects more defects than structural profiles.

Combining Substate and structural profiles yields better defect detection.

Substate Profiling is more practical and finer grained than existing state capture methods.

Abstract

Test suite reduction (TSR) aims at removing redundant test cases from regression test suites. A typical TSR approach ensures that structural profile elements covered by the original test suite are also covered by the reduced test suite. It is plausible that structural profiles might be unable to segregate failing runs from passing runs, which diminishes the effectiveness of TSR in regard to defect detection. This motivated us to explore state profiles, which are based on the collective values of program variables. This paper presents Substate Profiling, a new form of state profiling that enhances existing profile-based analysis techniques such as TSR and coverage-based fault localization. Compared to current approaches for capturing program states, Substate Profiling is more practical and finer grained. We evaluated our approach using thirteen multi-fault subject programs comprising 53 defects. Our study involved greedy TSR using Substate profiles and four structural profiles, namely, basic-block, branch, def-use pair, and the combination of the three. For the majority of the subjects, Substate Profiling detected considerably more defects with a comparable level of reduction. Also, Substate profiles were found to be complementary to structural profiles in many cases, thus, combining both types is beneficial.