Test case quality: an empirical study on belief and evidence

TL;DR

This study empirically investigates common beliefs about what makes a test case good, finding little evidence to support these beliefs despite their widespread acceptance in software engineering.

Contribution

It provides an empirical evaluation of eight hypotheses on test case quality, challenging assumptions based on developer beliefs.

Findings

No supporting evidence for the tested hypotheses.

Beliefs about test case characteristics may not correlate with effectiveness.

Highlights the need for evidence-based criteria in testing practices.

Abstract

Software testing is a mandatory activity in any serious software development process, as bugs are a reality in software development. This raises the question of quality: good tests are effective in finding bugs, but until a test case actually finds a bug, its effectiveness remains unknown. Therefore, determining what constitutes a good or bad test is necessary. This is not a simple task, and there are a number of studies that identify different characteristics of a good test case. A previous study evaluated 29 hypotheses regarding what constitutes a good test case, but the findings are based on developers' beliefs, which are subjective and biased. In this paper we investigate eight of these hypotheses, through an extensive empirical study based on open software repositories. Despite our best efforts, we were unable to find evidence that supports these beliefs. This indicates that, although these hypotheses represent good software engineering advice, they do not necessarily mean that they are enough to provide the desired outcome of good testing code.