EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learning

TL;DR

EvoIdeator introduces a reinforcement learning framework that uses structured, checklist-grounded feedback to systematically evolve scientific ideas, improving idea quality and generalization in autonomous knowledge discovery.

Contribution

The paper presents EvoIdeator, a novel RL approach that incorporates multi-dimensional and fine-grained feedback for scientific idea evolution, outperforming larger models.

Findings

EvoIdeator significantly outperforms larger models on scientific metrics.

The framework generalizes well to external feedback sources.

It enables scalable, self-refining autonomous ideation.

Abstract

Scientific idea generation is a cornerstone of autonomous knowledge discovery, yet the iterative evolution required to transform initial concepts into high-quality research proposals remains a formidable challenge for Large Language Models (LLMs). Existing Reinforcement Learning (RL) paradigms often rely on rubric-based scalar rewards that provide global quality scores but lack actionable granularity. Conversely, language-based refinement methods are typically confined to inference-time prompting, targeting models that are not explicitly optimized to internalize such critiques. To bridge this gap, we propose \textbf{EvoIdeator}, a framework that facilitates the evolution of scientific ideas by aligning the RL training objective with \textbf{checklist-grounded feedback}. EvoIdeator leverages a structured judge model to generate two synergistic signals: (1) \emph{lexicographic rewards} for multi-dimensional optimization, and (2) \emph{fine-grained language feedback} that offers span-level critiques regarding grounding, feasibility, and methodological rigor. By integrating these signals into the RL loop, we condition the policy to systematically utilize precise feedback during both optimization and inference. Extensive experiments demonstrate that EvoIdeator, built on Qwen3-4B, significantly outperforms much larger frontier models across key scientific metrics. Crucially, the learned policy exhibits strong generalization to diverse external feedback sources without further fine-tuning, offering a scalable and rigorous path toward self-refining autonomous ideation.