When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery

TL;DR

This paper demonstrates that binary feedback can outperform ordinal comparisons in ranking tasks, with theoretical proofs and empirical evidence showing faster convergence and better accuracy when binarizing ordinal data.

Contribution

It introduces a unified parametric framework for ordinal paired comparisons and proves that binarizing ordinal data can lead to more accurate ranking recovery.

Findings

Binarized data achieves faster exponential convergence in ranking error.

A specific pattern function minimizes SNR and enhances binarization benefits.

Empirical results on MovieLens support theoretical advantages of binary feedback.

Abstract

Paired comparison data, where users evaluate items in pairs, play a central role in ranking and preference learning tasks. While ordinal comparison data intuitively offer richer information than binary comparisons, this paper challenges that conventional wisdom. We propose a general parametric framework for modeling ordinal paired comparisons without ties. The model adopts a generalized additive structure, featuring a link function that quantifies the preference difference between two items and a pattern function that governs the distribution over ordinal response levels. This framework encompasses classical binary comparison models as special cases, by treating binary responses as binarized versions of ordinal data. Within this framework, we show that binarizing ordinal data can significantly improve the accuracy of ranking recovery. Specifically, we prove that under the counting algorithm, the ranking error associated with binary comparisons exhibits a faster exponential convergence rate than that of ordinal data. Furthermore, we characterize a substantial performance gap between binary and ordinal data in terms of a signal-to-noise ratio (SNR) determined by the pattern function. We identify the pattern function that minimizes the SNR and maximizes the benefit of binarization. Extensive simulations and a real application on the MovieLens dataset further corroborate our theoretical findings.