Automated Black-Box Boundary Value Detection

TL;DR

This paper introduces AutoBVA, an automated black-box method for detecting boundary values in software testing, using a program derivative metric to systematically identify and rank boundary candidates.

Contribution

It proposes a novel automated approach for boundary value detection that quantifies boundaryness and couples it with search algorithms for consistent results.

Findings

Effective boundary candidates identified with simple derivatives

Method works across diverse example programs

Broad sampling enhances boundary detection

Abstract

The input domain of software systems can typically be divided into sub-domains for which the outputs are similar. To ensure high quality it is critical to test the software on the boundaries between these sub-domains. Consequently, boundary value analysis and testing has been part of the toolbox of software testers for long and is typically taught early to students. However, despite its many argued benefits, boundary value analysis for a given specification or piece of software is typically described in abstract terms which allow for variation in how testers apply it. Here we propose an automated, black-box boundary value detection method to support software testers in systematic boundary value analysis with consistent results. The method builds on a metric to quantify the level of boundariness of test inputs: the program derivative. By coupling it with search algorithms we find and rank pairs of inputs as good boundary candidates, i.e. inputs close together but with outputs far apart. We implement our AutoBVA approach and evaluate it on a curated dataset of example programs. Our results indicate that even with a simple and generic program derivative variant in combination with broad sampling over the input space, interesting boundary candidates can be identified.