Parsimonious Mixed-Effects HodgeRank for Crowdsourced Preference Aggregation

TL;DR

This paper introduces a parsimonious mixed-effects HodgeRank model for crowdsourced preference aggregation, accounting for both common and personalized annotator behaviors, and demonstrates improved performance over traditional methods.

Contribution

It extends HodgeRank with a mixed-effects model incorporating random annotator effects and develops a novel algorithm for efficient estimation, enhancing preference aggregation accuracy.

Findings

Better test error performance than standard HodgeRank

Effectively captures individual annotator variations

Validated on simulated and real datasets

Abstract

In crowdsourced preference aggregation, it is often assumed that all the annotators are subject to a common preference or utility function which generates their comparison behaviors in experiments. However, in reality annotators are subject to variations due to multi-criteria, abnormal, or a mixture of such behaviors. In this paper, we propose a parsimonious mixed-effects model based on HodgeRank, which takes into account both the fixed effect that the majority of annotators follows a common linear utility model, and the random effect that a small subset of annotators might deviate from the common significantly and exhibits strongly personalized preferences. HodgeRank has been successfully applied to subjective quality evaluation of multimedia and resolves pairwise crowdsourced ranking data into a global consensus ranking and cyclic conflicts of interests. As an extension, our proposed methodology further explores the conflicts of interests through the random effect in annotator specific variations. The key algorithm in this paper establishes a dynamic path from the common utility to individual variations, with different levels of parsimony or sparsity on personalization, based on newly developed Linearized Bregman Algorithms with Inverse Scale Space method. Finally the validity of the methodology are supported by experiments with both simulated examples and three real-world crowdsourcing datasets, which shows that our proposed method exhibits better performance (i.e. smaller test error) compared with HodgeRank due to its parsimonious property.