LLMs Can Assist with Proposal Selection at Large User Facilities

TL;DR

This paper demonstrates that large language models can effectively assist in proposal selection at large user facilities by providing consistent, scalable rankings that correlate well with human judgments and reduce costs.

Contribution

The study introduces a novel application of LLMs for proposal ranking, showing they outperform traditional human review in consistency and cost-efficiency, and enable advanced proposal analysis.

Findings

LLMs' proposal rankings strongly correlate with human rankings (Spearman 0.2-0.8).

LLMs identify high-potential proposals as well as humans but at lower cost.

Embedding-based analysis of proposal similarity is feasible with LLMs.

Abstract

We explore how large language models (LLMs) can enhance the proposal selection process at large user facilities, offering a scalable, consistent, and cost-effective alternative to traditional human review. Proposal selection depends on assessing the relative strength among submitted proposals; however, traditional human scoring often suffers from weak inter-proposal correlations and is subject to reviewer bias and inconsistency. A pairwise preference-based approach is logically superior, providing a more rigorous and internally consistent basis for ranking, but its quadratic workload makes it impractical for human reviewers. We address this limitation using LLMs. Leveraging the uniquely well-curated proposals and publication records from three beamlines at the Spallation Neutron Source (SNS), Oak Ridge National Laboratory (ORNL), we show that the LLM rankings correlate strongly with the human rankings (Spearman $\rho\simeq 0.2-0.8$, improving to $\geq 0.5$ after 10\% outlier removal). Moreover, LLM performance is no worse than that of human reviewers in identifying proposals with high publication potential, while costing over two orders of magnitude less. Beyond ranking, LLMs enable advanced analyses that are challenging for humans, such as quantitative assessment of proposal similarity via embedding models, which provides information crucial for review committees.