Can Large Language Models Evaluate Grant Proposal Quality? Revisiting the Wenner{\aa}s and Wold Peer Review Data

TL;DR

This study evaluates the effectiveness of large language models in scoring grant proposals by comparing their assessments to expert peer review scores, revealing moderate correlations and potential practical uses despite weaker performance.

Contribution

First assessment of LLMs for scoring grant proposals, comparing their scores with expert reviews using historical data, and analyzing their potential role in funding decisions.

Findings

LLMs show moderate correlation among themselves (mean Spearman 0.34).

LLMs weakly correlate with expert scores (mean Spearman 0.22).

Best LLM correlation with experts was 0.33, about half of reviewer agreement.

Abstract

Purpose: Despite the importance of peer review for grant funding decisions, academics are often reluctant to conduct it. This can lead to long delays between submission and the final decision as well as the risk of substandard reviews from busy or non-specialist scholars. At least one funder now uses Large Language Models (LLMs) to reduce the reviewing burden but the accuracy of LLMs for scoring grant proposals needs to be assessed. Design/methodology/approach: This article compares scores from a range of medium sized open weights LLMs with peer review scores for a well-researched dataset, the Swedish Medical Council's post-doctoral fellowship applications from 1994. Findings: Whilst the LLM scores correlate moderately between each other (mean Spearman correlation: 0.34), they correlated weakly but positively and mostly statistically significantly with the average expert scores (mean Spearman correlation: 0.22). The highest rank correlation between expert scores and LLMs was 0.33 for Gemma 3 27b based on proposal titles and summaries without their main texts, which is about half (56%) of the correlation between reviewers. Research limitations: The small sample size, old funding call and heterogeneous evaluation criteria all undermine the robustness of the analysis. Practical implications: Despite the ability of LLMs to score grant proposals being quantitatively weaker than that of experts, at least in this special case, they may have role in application triage or tie-breaking. Originality/value: This is the first assessment of the value of LLM scores for funding proposals.