Vulnerability of Text-Matching in ML/AI Conference Reviewer Assignments to Collusions

TL;DR

This paper reveals vulnerabilities in ML/AI conference reviewer assignment systems that use text-matching, showing colluding reviewers can manipulate assignments even without bidding, and suggests improvements for robustness.

Contribution

It uncovers a new vulnerability in text-matching based reviewer assignment systems and proposes strategies to mitigate collusion risks.

Findings

Colluding reviewers can manipulate assignments without bidding.

Text similarity components are exploitable even in absence of bid manipulation.

Recommendations are provided to improve system robustness.

Abstract

In the peer review process of top-tier machine learning (ML) and artificial intelligence (AI) conferences, reviewers are assigned to papers through automated methods. These assignment algorithms consider two main factors: (1) reviewers' expressed interests indicated by their bids for papers, and (2) reviewers' domain expertise inferred from the similarity between the text of their previously published papers and the submitted manuscripts. A significant challenge these conferences face is the existence of collusion rings, where groups of researchers manipulate the assignment process to review each other's papers, providing positive evaluations regardless of their actual quality. Most efforts to combat collusion rings have focused on preventing bid manipulation, under the assumption that the text similarity component is secure. In this paper, we demonstrate that even in the absence of bidding, colluding reviewers and authors can exploit the machine learning based text-matching component of reviewer assignment used at top ML/AI venues to get assigned their target paper. We also highlight specific vulnerabilities within this system and offer suggestions to enhance its robustness.