Noise and fluctuations can undermine the efficiency of Majority Rule in Group Evaluation problems

TL;DR

This paper investigates how noise and fluctuations impact the effectiveness of majority rule in group evaluations, revealing that while majority rule is generally reliable, noise can significantly undermine its accuracy in certain scenarios.

Contribution

It introduces a game-theoretical, evolutionary framework to analyze the robustness of majority rule in noisy environments for crowdsourced evaluations.

Findings

Majority rule remains effective in most cases despite agents not being experts.

Noise can significantly reduce the reliability of majority-based evaluations.

Careful consideration of noise effects is essential for reliable group assessments.

Abstract

Crowdsourcing is a mechanism by means of which groups of people are able to execute a task by sharing ideas, efforts and resources. Thanks to the online technologies, crowdsourcing has become in the last decade an even more utilized process in different and diverse fields. An instance of such process is the so-called "label aggregation problem": in practice, it is the evaluation of an item by groups of agents, where each agent gives its own judgment of it. Starting from the individual evaluations of their members, how can the groups give their global assessment? In this work, by means of a game-theoretical, evolutionary approach, we show that in most cases the majority rule (the group evaluation is the evaluation of the majority of its members) is still the best way to get a reliable group evaluation, even when the agents are not the best experts of the topic at stake; on the other hand, we also show that noise (i.e., fortuitous errors, misunderstanding, or every possible source of non-deterministic outcomes) can undermine the efficiency of the procedure in non-trivial situations. Therefore, in order to make the process as reliable as possible, the presence of noise and its effects should be carefully taken into account.