RATE: Reviewer Profiling and Annotation-free Training for Expertise Ranking in Peer Review Systems

TL;DR

This paper introduces LR-bench, a new up-to-date benchmark for reviewer expertise evaluation, and RATE, a novel reviewer ranking framework that uses reviewer profiles and weak supervision to improve matching accuracy in peer review systems.

Contribution

The paper presents LR-bench, a high-fidelity dataset for reviewer expertise, and RATE, a new embedding-based ranking method that outperforms existing approaches without requiring annotated training data.

Findings

RATE achieves state-of-the-art performance on LR-bench and CMU datasets.

The approach outperforms strong embedding baselines.

Profiles based on recent publications improve reviewer matching accuracy.

Abstract

Reviewer assignment is increasingly critical yet challenging in the LLM era, where rapid topic shifts render many pre-2023 benchmarks outdated and where proxy signals poorly reflect true reviewer familiarity. We address this evaluation bottleneck by introducing LR-bench, a high-fidelity, up-to-date benchmark curated from 2024-2025 AI/NLP manuscripts with five-level self-assessed familiarity ratings collected via a large-scale email survey, yielding 1055 expert-annotated paper-reviewer-score annotations. We further propose RATE, a reviewer-centric ranking framework that distills each reviewer's recent publications into compact keyword-based profiles and fine-tunes an embedding model with weak preference supervision constructed from heuristic retrieval signals, enabling matching each manuscript against a reviewer profile directly. Across LR-bench and the CMU gold-standard dataset, our approach consistently achieves state-of-the-art performance, outperforming strong embedding baselines by a clear margin. We release LR-bench at https://huggingface.co/datasets/Gnociew/LR-bench, and a GitHub repository at https://github.com/Gnociew/RATE-Reviewer-Assign.