Improving Data Quality via Pre-Task Participant Screening in Crowdsourced GUI Experiments

TL;DR

This study introduces a pre-task screening method for crowdsourced GUI experiments that filters out careless participants, thereby improving data quality and the accuracy of performance models.

Contribution

It proposes a simple, effective pre-task that predicts participant reliability, enhancing data validity in crowdsourced GUI performance studies.

Findings

Pre-task screening reduces careless participant inclusion.

Tighter error thresholds improve model fit.

Data quality and model accuracy are significantly enhanced.

Abstract

In crowdsourced user experiments that collect performance data from graphical user interface (GUI) interactions, some participants ignore instructions or act carelessly, threatening the validity of performance models. We investigate a pre-task screening method that requires simple GUI operations analogous to the main task and uses the resulting error as a continuous quality signal. Our pre-task is a brief image-resizing task in which workers match an on-screen card to a physical card; workers whose resizing error exceeds a threshold are excluded from the main experiment. The main task is a standardized pointing experiment with well-established models of movement time and error rate. Across mouse- and smartphone-based crowdsourced experiments, we show that reducing the proportion of workers exhibiting unexpected behavior and tightening the pre-task threshold systematically improve the goodness of fit and predictive accuracy of GUI performance models, demonstrating that brief pre-task screening can enhance data quality.