Mathematical Modeling of Competitive Group Recommendation Systems with Application to Peer Review Systems

TL;DR

This paper develops a mathematical model for competitive group recommendation systems, particularly peer review, analyzing factors affecting accuracy and proposing strategies to optimize review processes and improve recommendation quality.

Contribution

It introduces a formal mathematical framework for peer review systems, analyzes review number and aggregation policies, and proposes a heterogeneous review strategy to enhance accuracy efficiently.

Findings

Three reviews per paper suffice for high accuracy at medium-tier conferences.

At least seven reviews per paper are needed for prestigious conferences.

A heterogeneous review strategy can improve accuracy by up to 30% with similar workload.

Abstract

In this paper, we present a mathematical model to capture various factors which may influence the accuracy of a competitive group recommendation system. We apply this model to peer review systems, i.e., conference or research grants review, which is an essential component in our scientific community. We explore number of important questions, i.e., how will the number of reviews per paper affect the accuracy of the overall recommendation? Will the score aggregation policy influence the final recommendation? How reviewers' preference may affect the accuracy of the final recommendation? To answer these important questions, we formally analyze our model. Through this analysis, we obtain the insight on how to design a randomized algorithm which is both computationally efficient and asymptotically accurate in evaluating the accuracy of a competitive group recommendation system. We obtain number of interesting observations: i.e., for a medium tier conference, three reviews per paper is sufficient for a high accuracy recommendation. For prestigious conferences, one may need at least seven reviews per paper to achieve high accuracy. We also propose a heterogeneous review strategy which requires equal or less reviewing workload, but can improve over a homogeneous review strategy in recommendation accuracy by as much as 30% . We believe our models and methodology are important building blocks to study competitive group recommendation systems.