Characterizing High-Quality Test Methods: A First Empirical Study

TL;DR

This paper empirically investigates the quality of individual test methods using mutation testing, revealing that high-quality methods are less affected by test smells but do not differ significantly in size or asserts.

Contribution

It introduces a method-level mutation testing approach to evaluate test method quality and provides empirical insights into factors distinguishing high- and low-quality test methods.

Findings

High-quality test methods are less affected by test smells.

No significant difference in size or number of asserts between high- and low-quality methods.

Test smells impact test method quality more than size or asserts.

Abstract

To assess the quality of a test suite, one can rely on mutation testing, which computes whether the overall test cases are adequately exercising the covered lines. However, this high level of granularity may overshadow the quality of individual test methods. In this paper, we propose an empirical study to assess the quality of test methods by relying on mutation testing at the method level. We find no major differences between high-quality and low-quality test methods in terms of size, number of asserts, and modifications. In contrast, high-quality test methods are less affected by critical test smells. Finally, we discuss practical implications for researchers and practitioners.