Rating through Voting: An Iterative Method for Robust Rating

TL;DR

This paper presents an iterative voting algorithm for robust ratings that resists collusion, bias, and randomness, and is effective for large-scale online rating systems, demonstrated through simulations and real-world data comparisons.

Contribution

The paper introduces a novel iterative voting-based rating method that decouples credibility assessment from ranking, enhancing robustness against manipulation and bias.

Findings

Ratings closely match IMDb scores

Differences align with Rotten Tomatoes critic ratings

High efficiency for large online systems

Abstract

In this paper we introduce an iterative voting algorithm and then use it to obtain a rating method which is very robust against collusion attacks as well as random and biased raters. Unlike the previous iterative methods, our method is not based on comparing submitted evaluations to an approximation of the final rating scores, and it entirely decouples credibility assessment of the cast evaluations from the ranking itself. The convergence of our algorithm relies on the existence of a fixed point of a continuous mapping which is also a stationary point of a constrained optimization objective. We have implemented and tested our rating method using both simulated data as well as real world data. In particular, we have applied our method to movie evaluations obtained from MovieLens and compared our results with IMDb and Rotten Tomatoes movie rating sites. Not only are the ratings provided by our system very close to IMDb rating scores, but when we differ from the IMDb ratings, the direction of such differences is essentially always towards the ratings provided by the critics in Rotten Tomatoes. Our tests demonstrate high efficiency of our method, especially for very large online rating systems, for which trust management is both of the highest importance and one of the most challenging problems.