Idea2Plan: Exploring AI-Powered Research Planning

TL;DR

This paper introduces the Idea2Plan benchmark to evaluate large language models' ability to convert research ideas into structured plans, revealing current strengths and limitations of models like GPT-5 in scientific research planning.

Contribution

The paper presents the first systematic benchmark for assessing LLMs' research planning capabilities, including a new dataset and evaluation methods for autonomous research support.

Findings

GPT-5 achieves the best performance on the benchmark

Substantial room for improvement remains in LLM research planning

The benchmark enables rigorous assessment of LLMs' research planning skills

Abstract

Large language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields. In this work, we investigate how LLMs can handle the transition from conceptual research ideas to well-structured research plans. Effective research planning not only supports scientists in advancing their research but also represents a crucial capability for the development of autonomous research agents. Despite its importance, the field lacks a systematic understanding of LLMs' research planning capability. To rigorously measure this capability, we introduce the Idea2Plan task and Idea2Plan Bench, a benchmark built from 200 ICML 2025 Spotlight and Oral papers released after major LLM training cutoffs. Each benchmark instance includes a research idea and a grading rubric capturing the key components of valid plans. We further propose Idea2Plan JudgeEval, a complementary benchmark to assess the reliability of LLM-based judges against expert annotations. Experimental results show that GPT-5 and GPT-5-mini achieve the strongest performance on the benchmark, though substantial headroom remains for future improvement. Our study provides new insights into LLMs' capability for research planning and lay the groundwork for future progress.