The Effect of Class Imbalance and Order on Crowdsourced Relevance Judgments

TL;DR

This study investigates how class imbalance and presentation order affect crowd workers' relevance judgments, revealing that showing relevant documents before non-relevant ones improves judgment quality.

Contribution

It provides empirical evidence on the impact of data order on crowd-sourced relevance assessments, highlighting a simple method to enhance judgment accuracy.

Findings

Presenting relevant documents first improves judgment quality

Class imbalance influences worker efficiency and effectiveness

Order of presentation affects bias in relevance judgments

Abstract

In this paper we study the effect on crowd worker efficiency and effectiveness of the dominance of one class in the data they process. We aim at understanding if there is any positive or negative bias in workers seeing many negative examples in the identification of positive labels. To test our hypothesis, we design an experiment where crowd workers are asked to judge the relevance of documents presented in different orders. Our findings indicate that there is a significant improvement in the quality of relevance judgements when presenting relevant results before the non-relevant ones.