Measuring Software Testability via Automatically Generated Test Cases

TL;DR

This paper introduces a novel method for estimating software testability using automatic test case generation and mutation analysis, providing a more comprehensive measure than traditional metrics.

Contribution

It proposes a new approach that combines automatic test generation and mutation analysis to better estimate software testability, validated through experiments.

Findings

Testability estimates complement traditional metrics.

Combining methods improves prediction accuracy.

The approach offers a new dimension for assessing testability.

Abstract

Estimating software testability can crucially assist software managers to optimize test budgets and software quality. In this paper, we propose a new approach that radically differs from the traditional approach of pursuing testability measurements based on software metrics, e.g., the size of the code or the complexity of the designs. Our approach exploits automatic test generation and mutation analysis to quantify the evidence about the relative hardness of developing effective test cases. In the paper, we elaborate on the intuitions and the methodological choices that underlie our proposal for estimating testability, introduce a technique and a prototype that allows for concretely estimating testability accordingly, and discuss our findings out of a set of experiments in which we compare the performance of our estimations both against and in combination with traditional software metrics. The results show that our testability estimates capture a complementary dimension of testability that can be synergistically combined with approaches based on software metrics to improve the accuracy of predictions.