AIRA_2: Overcoming Bottlenecks in AI Research Agents

TL;DR

AIRA$_2$ introduces architectural innovations like asynchronous multi-GPU execution, reliable evaluation protocols, and dynamic LLM operators to overcome key bottlenecks in AI research agents, significantly improving performance and scalability.

Contribution

The paper presents AIRA$_2$, a novel AI research agent architecture that addresses existing bottlenecks through three key components, leading to superior performance and scalability.

Findings

AIRA$_2$ achieves 81.5% percentile rank at 24 hours on MLE-bench-30.

AIRA$_2$ surpasses human state-of-the-art on 6 out of 20 tasks in AIRS-Bench.

Each architectural component of AIRA$_2$ is necessary for its performance gains.

Abstract

Existing research has identified three structural performance bottlenecks in AI research agents: (1) synchronous single-GPU execution constrains sample throughput, limiting the benefit of search; (2) a generalization gap where validation-based selection causes overfitting and performance to degrade over extended search horizons; and (3) the limited capability of fixed, single-turn LLM operators imposes a ceiling on search performance. We introduce AIRA$_2$, which addresses these bottlenecks through three architectural choices: an asynchronous multi-GPU worker pool that increases experiment throughput linearly; a Hidden Consistent Evaluation protocol that delivers a reliable evaluation signal; and ReAct agents that dynamically scope their actions and debug interactively. On MLE-bench-30, AIRA$^{\dagger}_{2}$ achieves a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, outperforming the strongest baseline, which achieves 72.7%. On AIRS-Bench, AIRA$_2$ exceeds human state-of-the-art on 6 out of 20 diverse research tasks. Ablations confirm that each architectural component is necessary, that performance follows a predictable scaling law that transfers across LLM backbones, and that the "overfitting" reported in prior work was driven by evaluation noise rather than true data memorization.