Test cases as a measurement instrument in experimentation

TL;DR

This paper investigates how different test suite construction methods impact measurement accuracy and experimental results in software engineering, revealing significant variability and emphasizing the need for comprehensive disclosure for reproducibility.

Contribution

It demonstrates that test suite construction methods significantly influence measurement outcomes and advocates for full disclosure of experimental materials to improve reproducibility.

Findings

Test suite construction causes up to +-60% variation in response variables.

Different test suites lead to inconsistent experimental results.

Reproducibility is hindered by insufficient disclosure of datasets and analysis code.

Abstract

Background: Test suites are frequently used to quantify relevant software attributes, such as quality or productivity. Problem: We have detected that the same response variable, measured using different test suites, yields different experiment results. Aims: Assess to which extent differences in test case construction influence measurement accuracy and experimental outcomes. Method: Two industry experiments have been measured using two different test suites, one generated using an ad-hoc method and another using equivalence partitioning. The accuracy of the measures has been studied using standard procedures, such as ISO 5725, Bland-Altman and Interclass Correlation Coefficients. Results: There are differences in the values of the response variables up to +-60%, depending on the test suite (ad-hoc vs. equivalence partitioning) used. Conclusions: The disclosure of datasets and analysis code is insufficient to ensure the reproducibility of SE experiments. Experimenters should disclose all experimental materials needed to perform independent measurement and re-analysis.