InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research

TL;DR

InnovatorBench is a comprehensive benchmark platform designed to evaluate AI agents' capabilities in conducting end-to-end large language model research tasks, highlighting current limitations and potential areas for improvement.

Contribution

The paper introduces InnovatorBench and ResearchGym, enabling realistic assessment of LLM research agents and revealing the challenges frontier models face in complex, long-horizon tasks.

Findings

Frontier models show promise in code-driven tasks.

Models struggle with fragile algorithms and long-horizon decision making.

Agents need over 11 hours to reach peak performance on the benchmark.

Abstract

AI agents could accelerate scientific discovery by automating hypothesis formation, experiment design, coding, execution, and analysis, yet existing benchmarks probe narrow skills in simplified settings. To address this gap, we introduce InnovatorBench, a benchmark-platform pair for realistic, end-to-end assessment of agents performing Large Language Model (LLM) research. It comprises 20 tasks spanning Data Construction, Filtering, Augmentation, Loss Design, Reward Design, and Scaffold Construction, which require runnable artifacts and assessment of correctness, performance, output quality, and uncertainty. To support agent operation, we develop ResearchGym, a research environment offering rich action spaces, distributed and long-horizon execution, asynchronous monitoring, and snapshot saving. We also implement a lightweight ReAct agent that couples explicit reasoning with executable planning using frontier models such as Claude-4, GPT-5, GLM-4.5, and Kimi-K2. Our experiments demonstrate that while frontier models show promise in code-driven research tasks, they struggle with fragile algorithm-related tasks and long-horizon decision making, such as impatience, poor resource management, and overreliance on template-based reasoning. Furthermore, agents require over 11 hours to achieve their best performance on InnovatorBench, underscoring the benchmark's difficulty and showing the potential of InnovatorBench to be the next generation of code-based research benchmark.