The Search for the Laws of Automatic Random Testing

TL;DR

This paper explores the possibility of establishing a mathematical law for estimating remaining software faults through automated random testing, potentially aiding project management and theoretical understanding.

Contribution

It proposes a poly-logarithmic law for fault estimation based on empirical analysis of automated testing data, advancing the understanding of software testing laws.

Findings

Suggests a poly-logarithmic law for fault estimation

Based on analysis of automated tests with contracts

Requires further validation on diverse code bases

Abstract

Can one estimate the number of remaining faults in a software system? A credible estimation technique would be immensely useful to project managers as well as customers. It would also be of theoretical interest, as a general law of software engineering. We investigate possible answers in the context of automated random testing, a method that is increasingly accepted as an effective way to discover faults. Our experimental results, derived from best-fit analysis of a variety of mathematical functions, based on a large number of automated tests of library code equipped with automated oracles in the form of contracts, suggest a poly-logarithmic law. Although further confirmation remains necessary on different code bases and testing techniques, we argue that understanding the laws of testing may bring significant benefits for estimating the number of detectable faults and comparing different projects and practices.