Traditional statistical representations outperform generative AI in identifying expert peer reviewers

TL;DR

Traditional statistical methods outperform large language models like GPT-4 in accurately identifying domain experts for peer review, highlighting the importance of fine-grained vocabulary in expertise detection.

Contribution

This study provides a rigorous empirical evaluation comparing statistical and AI-driven methods for expert identification, demonstrating the superiority of traditional statistical representations.

Findings

Term Frequency-Inverse Document Frequency identified experts 79.5% of the time within top 25 recommendations.

GPT-4o mini achieved 51.5% success in expert identification.

Statistical methods outperform generative AI in specialized scientific tasks.

Abstract

The exponential growth of scientific submissions has strained the peer review system. Despite the rapidly expanding global pool of researchers, this unprecedented scale has rendered the previous approach of manual expert identification unfeasible. Therefore, institutions have naturally turned to Large Language Models (LLMs) to automate intricate processes like expert reviewer identification. However, the reliability of these new models in accurately identifying domain experts lacks rigorous evaluation. We conduct a comprehensive empirical evaluation of statistical and AI-driven expertise identification methodologies to benchmark their reliability and limitations. Framing expert identification as an information retrieval problem, we utilize the distributed peer review system of a major international astronomical observatory, where proposal authorship serves as our proxy ground truth for domain expertise. Evaluating six retrieval methodologies utilized across observatories and computer science conferences, we demonstrate that traditional statistical representations outperform generative AI. Specifically, Term Frequency-Inverse Document Frequency successfully identified a labeled expert within the top 25 recommendations 79.5% of the time, compared to 51.5% for GPT-4o mini. Our results highlight that distinguishing subfield expertise requires fine-grained vocabulary, which is obscured by the semantic smoothing in generative methods. By establishing a rigorous evaluation framework for automated peer review, we demonstrate that transparent and reproducible statistical representations still outperform computationally expensive LLMs in specialized scientific tasks.