Distributional Ground Truth: Non-Redundant Crowdsourcing Data Quality Control in UI Labeling Tasks

TL;DR

This paper introduces a non-redundant crowdsourcing quality control method for UI labeling tasks that predicts worker accuracy using distributional homogeneity, reducing the need for outcome redundancy.

Contribution

The paper proposes a novel distributional ground truth approach based on the Kolmogorov-Smirnov test for quality prediction in crowdsourced UI labeling, eliminating the need for redundant labeling.

Findings

Achieves R2 over 0.8 with 17-27% trusted set size

Outperforms baseline mean Time-on-Task model

Reduces work effort and costs

Abstract

HCI increasingly employs Machine Learning and Image Recognition, in particular for visual analysis of user interfaces (UIs). A popular way for obtaining human-labeled training data is Crowdsourcing, typically using the quality control methods ground truth and majority consensus, which necessitate redundancy in the outcome. In our paper we propose a non-redundant method for prediction of crowdworkers' output quality in web UI labeling tasks, based on homogeneity of distributions assessed with two-sample Kolmogorov-Smirnov test. Using a dataset of about 500 screenshots with over 74,000 UI elements located and classified by 11 trusted labelers and 298 Amazon Mechanical Turk crowdworkers, we demonstrate the advantage of our approach over the baseline model based on mean Time-on-Task. Exploring different dataset partitions, we show that with the trusted set size of 17-27% UIs our "distributional ground truth" model can achieve R2s of over 0.8 and help to obviate the ancillary work effort and expenses.