On the impact of predicate complexity in crowdsourced classification tasks

TL;DR

This paper investigates how the complexity of predicate formulation in crowdsourced classification tasks affects performance, providing empirical insights to guide task design in micro-task markets.

Contribution

It offers empirical evidence on the impact of predicate complexity on classification accuracy, highlighting the importance of task formulation strategies in crowdsourcing.

Findings

Predicate complexity significantly influences classification performance.

Different predicate formulation strategies yield varying accuracy levels.

Combining crowd workers with machine learning classifiers improves results.

Abstract

This paper explores and offers guidance on a specific and relevant problem in task design for crowdsourcing: how to formulate a complex question used to classify a set of items. In micro-task markets, classification is still among the most popular tasks. We situate our work in the context of information retrieval and multi-predicate classification, i.e., classifying a set of items based on a set of conditions. Our experiments cover a wide range of tasks and domains, and also consider crowd workers alone and in tandem with machine learning classifiers. We provide empirical evidence into how the resulting classification performance is affected by different predicate formulation strategies, emphasizing the importance of predicate formulation as a task design dimension in crowdsourcing.