The Benefits of Diversity: Combining Comparisons and Ratings for Efficient Scoring

TL;DR

This paper introduces SCoRa, a probabilistic model that combines comparisons and ratings for more accurate and robust scoring, outperforming single-method approaches especially when top entity ordering is crucial.

Contribution

The paper presents SCoRa, a novel unified model that effectively integrates comparison and rating signals for preference learning, with proven theoretical guarantees and empirical advantages.

Findings

SCoRa recovers accurate scores even with model mismatch.

Combining comparisons and ratings outperforms individual methods in key settings.

SCoRa provides a versatile foundation for preference learning.

Abstract

Should humans be asked to evaluate entities individually or comparatively? This question has been the subject of long debates. In this work, we show that, interestingly, combining both forms of preference elicitation can outperform the focus on a single kind. More specifically, we introduce SCoRa (Scoring from Comparisons and Ratings), a unified probabilistic model that allows to learn from both signals. We prove that the MAP estimator of SCoRa is well-behaved. It verifies monotonicity and robustness guarantees. We then empirically show that SCoRa recovers accurate scores, even under model mismatch. Most interestingly, we identify a realistic setting where combining comparisons and ratings outperforms using either one alone, and when the accurate ordering of top entities is critical. Given the de facto availability of signals of multiple forms, SCoRa additionally offers a versatile foundation for preference learning.