The effect of discrete vs. continuous-valued ratings on reputation and ranking systems

TL;DR

This paper investigates how the use of discrete versus continuous rating scales affects the performance of reputation and ranking algorithms, revealing that lower resolution can sometimes enhance system accuracy due to increased noise.

Contribution

It provides a comparative analysis of co-determination algorithms under different rating scales, highlighting the significant impact of rating resolution on their effectiveness.

Findings

Lower rating resolution impacts algorithm performance significantly.

Increased noise from low-resolution ratings can improve overall system accuracy.

Discrete rating scales can sometimes outperform continuous ones in reputation systems.

Abstract

When users rate objects, a sophisticated algorithm that takes into account ability or reputation may produce a fairer or more accurate aggregation of ratings than the straightforward arithmetic average. Recently a number of authors have proposed different co-determination algorithms where estimates of user and object reputation are refined iteratively together, permitting accurate measures of both to be derived directly from the rating data. However, simulations demonstrating these methods' efficacy assumed a continuum of rating values, consistent with typical physical modelling practice, whereas in most actual rating systems only a limited range of discrete values (such as a 5-star system) is employed. We perform a comparative test of several co-determination algorithms with different scales of discrete ratings and show that this seemingly minor modification in fact has a significant impact on algorithms' performance. Paradoxically, where rating resolution is low, increased noise in users' ratings may even improve the overall performance of the system.