Evaluating LLM-Based Grant Proposal Review via Structured Perturbations

TL;DR

This study assesses the effectiveness of large language models in grant proposal review by applying structured perturbations to evaluate sensitivity and reliability across different review architectures and quality axes.

Contribution

It introduces a perturbation-based framework for evaluating LLM review sensitivity and compares review architectures, highlighting the strengths of section-level analysis.

Findings

Section-level review outperforms other architectures in detection and reliability.

Alignment issues are easily detected, but clarity flaws are often missed.

LLMs provide valid but skewed feedback, with high variability in performance.

Abstract

As AI-assisted grant proposals outpace manual review capacity in a kind of ``Malthusian trap'' for the research ecosystem, this paper investigates the capabilities and limitations of LLM-based grant reviewing for high-stakes evaluation. Using six EPSRC proposals, we develop a perturbation-based framework probing LLM sensitivity across six quality axes: funding, timeline, competency, alignment, clarity, and impact. We compare three review architectures: single-pass review, section-by-section analysis, and a 'Council of Personas' ensemble emulating expert panels. The section-level approach significantly outperforms alternatives in both detection rate and scoring reliability, while the computationally expensive council method performs no better than baseline. Detection varies substantially by perturbation type, with alignment issues readily identified but clarity flaws largely missed by all systems. Human evaluation shows LLM feedback is largely valid but skewed toward compliance checking over holistic assessment. We conclude that current LLMs may provide supplementary value within EPSRC review but exhibit high variability and misaligned review priorities. We release our code and any non-protected data.