Intrinsic Properties of Complete Test Suites

TL;DR

This paper explores the concept of completeness in test suites, especially considering blocking test cases, and establishes bounds on implementation states for test suite completeness.

Contribution

It introduces a new notion of completeness for blocking test cases and characterizes test suite completeness under these conditions.

Findings

Defined a new notion of completeness for blocking test cases.

Established an upper bound on implementation states for test suite completeness.

Characterized test suite completeness in both classical and new blocking scenarios.

Abstract

Completeness is a desirable property of test suites. Roughly, completeness guarantees that a non-equivalent implementation under test will always be identified. Several approaches proposed sufficient, and sometimes also necessary, conditions on the specification model and on the test suite in order to guarantee completeness. Usually, these approaches impose several restrictions on the specification and on the implementations, such as requiring them to be reduced or complete. Further, test cases are required to be non-blocking --- that is, they must run to completion --- on both the specification and the implementation models. In this work we deal test cases that can be blocking, we define a new notion that captures completeness, and we characterize test suite completeness in this new scenario. We establish an upper bound on the number of states of implementations beyond which no test suite can be complete, both in the classical sense and in the new scenario with blocking test cases.