OmniReview: A Large-scale Benchmark and LLM-enhanced Framework for Realistic Reviewer Recommendation

TL;DR

This paper introduces OmniReview, a large-scale, multi-source dataset and a novel LLM-enhanced framework called Pro-MMoE for realistic reviewer recommendation, addressing data scarcity and interpretability issues in peer review systems.

Contribution

It provides a comprehensive dataset and a new multi-gate MMoE framework that leverages LLMs for improved reviewer recommendation accuracy and interpretability.

Findings

Pro-MMoE outperforms existing methods on six of seven metrics.

The dataset includes over 202,000 verified review records from multiple sources.

The hierarchical evaluation framework better reflects real-world editorial workflows.

Abstract

Academic peer review remains the cornerstone of scholarly validation, yet the field faces some challenges in data and methods. From the data perspective, existing research is hindered by the scarcity of large-scale, verified benchmarks and oversimplified evaluation metrics that fail to reflect real-world editorial workflows. To bridge this gap, we present OmniReview, a comprehensive dataset constructed by integrating multi-source academic platforms encompassing comprehensive scholarly profiles through the disambiguation pipeline, yielding 202, 756 verified review records. Based on this data, we introduce a three-tier hierarchical evaluaion framework to assess recommendations from recall to precise expert identification. From the method perspective, existing embedding-based approaches suffer from the information bottleneck of semantic compression and limited interpretability. To resolve these method limitations, we propose Profiling Scholars with Multi-gate Mixture-of-Experts (Pro-MMoE), a novel framework that synergizes Large Language Models (LLMs) with Multi-task Learning. Specifically, it utilizes LLM-generated semantic profiles to preserve fine-grained expertise nuances and interpretability, while employing a Task-Adaptive MMoE architecture to dynamically balance conflicting evaluation goals. Comprehensive experiments demonstrate that Pro-MMoE achieves state-of-the-art performance across six of seven metrics, establishing a new benchmark for realistic reviewer recommendation.