Towards Execution-Grounded Automated AI Research

TL;DR

This paper explores the feasibility of automated execution in AI research, demonstrating that execution-guided methods can effectively improve research ideas and optimize training processes, with analysis of their limitations and potential.

Contribution

It introduces an automated executor for implementing AI research ideas, evaluates execution-guided search and reinforcement learning, and analyzes their effectiveness and limitations.

Findings

Execution-guided evolutionary search outperforms baselines in efficiency and results.

Automated executor successfully implements a large fraction of ideas from frontier LLMs.

Reinforcement learning improves average reward but suffers from mode collapse.

Abstract

Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems - LLM pre-training and post-training - into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline (69.4% vs 48.0%) on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts towards execution-grounded automated AI research.