Capability-Aware Early-Stage Research Idea Evaluation

TL;DR

This paper introduces a capability-aware framework that predicts research paper acceptance and ratings at early stages using only author info and research ideas, enhancing prediction accuracy without full text or results.

Contribution

It presents a novel three-way transformer architecture that integrates author info, capability presentation, and research ideas for early-stage research outcome prediction.

Findings

Outperforms single-way models like finetuned BERT variants.

Capability prediction significantly improves overall accuracy.

Applicable to early-stage research outcome prediction and resource allocation.

Abstract

Predicting the outcomes of research ideas at their conceptual stage (i.e. before significant resources are committed) holds great potential for optimizing scientific resource allocation and research planning. While existing methods rely heavily on finished manuscripts or peer reviews, we propose a novel capability-aware framework that predicts paper acceptance and ratings using only author information and research ideas, without requiring full text or experimental results. Our approach integrates author information, (inferred) capability presentation, and research ideas through a three-way transformer architecture with flexible fusion mechanisms. We also introduce a two-stage architecture for learning the capability representation given the author information and idea. Experiments show that our method significantly outperform the single-way models by finetuning bert-base and bert-large, and the capability predicting significantly increase the predictive accuracy of the final model. The proposed method can be applied in both early-stage research outcome prediction and scientific resource allocation.