The Majority Vote Paradigm Shift: When Popular Meets Optimal

TL;DR

This paper investigates the conditions under which the simple Majority Vote method is theoretically optimal for label aggregation in crowdsourcing, providing a principled approach to model selection based on annotation noise limits.

Contribution

It characterizes the conditions where Majority Vote achieves optimal label estimation, offering a theoretical foundation and practical criteria for its effective use.

Findings

Majority Vote is optimal under specific noise conditions.

Theoretical bounds for label recovery are established.

Experimental results validate the theoretical analysis.

Abstract

Reliably labelling data typically requires annotations from multiple human workers. However, humans are far from being perfect. Hence, it is a common practice to aggregate labels gathered from multiple annotators to make a more confident estimate of the true label. Among many aggregation methods, the simple and well known Majority Vote (MV) selects the class label polling the highest number of votes. However, despite its importance, the optimality of MV's label aggregation has not been extensively studied. We address this gap in our work by characterising the conditions under which MV achieves the theoretically optimal lower bound on label estimation error. Our results capture the tolerable limits on annotation noise under which MV can optimally recover labels for a given class distribution. This certificate of optimality provides a more principled approach to model selection for label aggregation as an alternative to otherwise inefficient practices that sometimes include higher experts, gold labels, etc., that are all marred by the same human uncertainty despite huge time and monetary costs. Experiments on both synthetic and real world data corroborate our theoretical findings.