Exploratory Testing: One Size Doesn't Fit All

TL;DR

This paper introduces a scale for different levels of exploratory testing, demonstrating how varying degrees of exploration can improve software testing effectiveness through empirical focus group evaluations.

Contribution

It proposes a classification scale for ET levels based on test charter formulation, filling a gap in understanding ET diversity and guiding structured testing approaches.

Findings

Distinct ET levels have identifiable characteristics

Combining ET levels can enhance testing effectiveness

The classification aids in structuring test charters

Abstract

Exploratory testing (ET) is a powerful and efficient way of testing software by integrating design, execution, and analysis of tests during a testing session. ET is often contrasted with scripted testing, and seen as a choice between black and white. We pose that there are different levels of exploratory testing from fully exploratory to fully scripted and propose a scale for the degree of exploration for ET. The degree is defined through levels of ET, which correspond to the way test charters are formulated. We have evaluated the classification through focus groups at four companies and identified factors that influence the level of exploratory testing. The results show that the proposed ET levels have distinguishing characteristics and that the levels can be used as a guide to structure test charters. Our study also indicates that applying a combination of ET levels can be beneficial in achieving effective testing.