Toward a Robust Crowd-labeling Framework using Expert Evaluation and Pairwise Comparison

TL;DR

This paper introduces ELICE, a new crowd-labeling framework that integrates expert evaluation and pairwise comparison to improve label accuracy and delay phase transition effects caused by low-quality labelers.

Contribution

The paper proposes ELICE, a novel framework that combines expert labels with crowd labels to enhance accuracy and robustness in large-scale data labeling tasks.

Findings

ELICE outperforms existing methods in accuracy.

A lower bound on expert-labeled instances needed is derived.

ELICE effectively delays phase transition in label quality.

Abstract

Crowd-labeling emerged from the need to label large-scale and complex data, a tedious, expensive, and time-consuming task. One of the main challenges in the crowd-labeling task is to control for or determine in advance the proportion of low-quality/malicious labelers. If that proportion grows too high, there is often a phase transition leading to a steep, non-linear drop in labeling accuracy as noted by Karger et al. [2014]. To address these challenges, we propose a new framework called Expert Label Injected Crowd Estimation (ELICE) and extend it to different versions and variants that delay phase transition leading to a better labeling accuracy. ELICE automatically combines and boosts bulk crowd labels supported by labels from experts for limited number of instances from the dataset. The expert-labels help to estimate the individual ability of crowd labelers and difficulty of each instance, both of which are used to aggregate the labels. Empirical evaluation shows the superiority of ELICE as compared to other state-of-the-art methods. We also derive a lower bound on the number of expert-labeled instances needed to estimate the crowd ability and dataset difficulty as well as to get better quality labels.