Crowdtesting : When is The Party Over?

TL;DR

This paper introduces automated decision support methods to determine optimal closure times for crowdtesting tasks, improving bug detection efficiency and reducing costs in dynamic, uncertain environments.

Contribution

It proposes and evaluates eight novel models for predicting crowdtesting task closure, addressing a critical gap in current crowdtesting management practices.

Findings

Median of 91% bugs detected

49% cost savings achieved

Effective prediction models developed

Abstract

Trade-offs such as "how much testing is enough" are critical yet challenging project decisions in software engineering. Most existing approaches adopt risk-driven or value-based analysis to prioritize test cases and minimize test runs. However, none of these is applicable to the emerging crowd testing paradigm where task requesters typically have no control over online crowdworkers's dynamic behavior and uncertain performance. In current practice, deciding when to close a crowdtesting task is largely done by guesswork due to lack of decision support. This paper intends to fill this gap by introducing automated decision support for monitoring and determining appropriate time to close the crowdtesting tasks. First, this paper investigates the necessity and feasibility of close prediction of crowdtesting tasks based on industrial dataset. Then,it designs 8 methods for close prediction, based on various models including the bug trend, bug arrival model, capture-recapture model.Finally, the evaluation is conducted on 218 crowdtesting tasks from one of the largest crowdtesting platforms in China, and the results show that a median of 91% bugs can be detected with 49% saved cost.