A Meta-evaluation of Scientific Research Proposals: Different Ways of Comparing Rejected to Awarded Applications

TL;DR

This study compares funding decisions with scientometric indicators across life and social sciences, revealing that awarded applicants generally perform better, but top rejected applicants can outperform awardees in citation impact, with field-specific productivity differences.

Contribution

It introduces a meta-evaluation approach comparing rejected and awarded applications using multiple data sets and scientometric metrics across two scientific fields.

Findings

Awarded applicants perform better overall than rejected ones.

Top rejected applicants can outperform awardees in citation impact.

Productivity differences vary between life sciences and social sciences.

Abstract

Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life sciences. In both fields, awarded applicants perform on average better than all rejected applicants. If only the most preeminent rejected applicants are considered in both fields, they score better than the awardees on citation impact. With regard to productivity we find differences between the fields: While the awardees in life sciences outperform on average the most preeminent rejected applicants, the situation is reversed in social sciences.