Recall, Risk, and Governance in Automated Proposal Screening for Research Funding: Evidence from a National Funding Programme

TL;DR

This study empirically compares rule-based and LLM-based automated proposal screening methods in a national research funding context, highlighting the importance of error profiles and institutional suitability for high-stakes decisions.

Contribution

It provides the first empirical comparison of automated screening approaches against committee decisions, emphasizing error asymmetry and institutional context in AI tool evaluation.

Findings

TF-IDF approach outperforms LLM in recall and false negatives

LLM-based system excludes more proposals, risking irrecoverable errors

Error profile and transparency are crucial for AI suitability in funding decisions

Abstract

Research funding agencies are increasingly exploring automated tools to support early-stage proposal screening. Recent advances in large language models (LLMs) have generated optimism regarding their use for text-based evaluation, yet their institutional suitability for high-stakes screening decisions remains underexplored. In particular, there is limited empirical evidence on how automated screening systems perform when evaluated against institutional error costs. This study compares two automated approaches for proposal screening against the priorities of a national funding call: A transparent, rule-based method using term frequency-inverse document frequency (TF-IDF) with domain-specific keyword engineering, and a semantic classification approach based on a large language model. Using selection committee decisions as ground truth for 959 proposals, we evaluate performance with particular attention to error structure. The results show that the TF-IDF-based approach outperforms the LLM-based system across standard metrics, achieving substantially higher recall (78.95\% vs 45.82\%) and producing far fewer false negatives (68 vs 175). The LLM-based system excludes more than half of the proposals ultimately selected by the committee. While false positives can be corrected through subsequent peer review, false negatives represent an irrecoverable exclusion from expert evaluation. By foregrounding error asymmetry and institutional context, this study demonstrates that the suitability of automated screening systems depends not on model sophistication alone, but on how their error profiles, transparency, and auditability align with research evaluation practice. These findings suggest that evaluation design and error tolerance should guide the use of AI-assisted screening tools in research funding more broadly.