Test cases as a measurement instrument in experimentation
Oscar Dieste, Fernando Uyaguari, Sira Vegas, Natalia Juristo

TL;DR
This paper investigates how different test suite construction methods impact measurement accuracy and experimental results in software engineering, revealing significant variability and emphasizing the need for comprehensive disclosure for reproducibility.
Contribution
It demonstrates that test suite construction methods significantly influence measurement outcomes and advocates for full disclosure of experimental materials to improve reproducibility.
Findings
Test suite construction causes up to +-60% variation in response variables.
Different test suites lead to inconsistent experimental results.
Reproducibility is hindered by insufficient disclosure of datasets and analysis code.
Abstract
Background: Test suites are frequently used to quantify relevant software attributes, such as quality or productivity. Problem: We have detected that the same response variable, measured using different test suites, yields different experiment results. Aims: Assess to which extent differences in test case construction influence measurement accuracy and experimental outcomes. Method: Two industry experiments have been measured using two different test suites, one generated using an ad-hoc method and another using equivalence partitioning. The accuracy of the measures has been studied using standard procedures, such as ISO 5725, Bland-Altman and Interclass Correlation Coefficients. Results: There are differences in the values of the response variables up to +-60%, depending on the test suite (ad-hoc vs. equivalence partitioning) used. Conclusions: The disclosure of datasets and analysis…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Software Testing and Debugging Techniques · Software Reliability and Analysis Research
