Designing Optimal Binary Rating Systems

TL;DR

This paper develops a theoretical framework and an efficient algorithm for designing optimal binary rating systems that quickly recover true item rankings, validated through real-world experiments.

Contribution

It formalizes the performance measure for binary rating systems, introduces an algorithm for optimal design, and demonstrates practical implementation with empirical validation.

Findings

Optimal binary rating systems recover true rankings efficiently.

The proposed algorithm computes near-optimal rating strategies.

Empirical validation confirms the effectiveness of the designed system.

Abstract

Modern online platforms rely on effective rating systems to learn about items. We consider the optimal design of rating systems that collect binary feedback after transactions. We make three contributions. First, we formalize the performance of a rating system as the speed with which it recovers the true underlying ranking on items (in a large deviations sense), accounting for both items' underlying match rates and the platform's preferences. Second, we provide an efficient algorithm to compute the binary feedback system that yields the highest such performance. Finally, we show how this theoretical perspective can be used to empirically design an implementable, approximately optimal rating system, and validate our approach using real-world experimental data collected on Amazon Mechanical Turk.