Towards Evidence-based Testability Measurements

TL;DR

This paper introduces a new evidence-based approach to measuring software testability using automatic test generation and mutation analysis, moving beyond traditional metric-based methods.

Contribution

It proposes two novel testability metrics derived from evidence, along with a prototype implementation and initial validation results.

Findings

Metrics reflect actual testability issues

Prototype successfully computes the proposed metrics

Initial results show promise in assessing testability

Abstract

Evaluating Software testability can assist software managers in optimizing testing budgets and identifying opportunities for refactoring. In this paper, we abandon the traditional approach of pursuing testability measurements based on the correlation between software metrics and test characteristics observed on past projects, e.g., the size, the organization or the code coverage of the test cases. We propose a radically new approach that exploits automatic test generation and mutation analysis to quantify the amount of evidence about the relative hardness of identifying effective test cases. We introduce two novel evidence-based testability metrics, describe a prototype to compute them, and discuss initial findings on whether our measurements can reflect actual testability issues.