AlphaLab: Autonomous Multi-Agent Research Across Optimization Domains with Frontier LLMs

TL;DR

AlphaLab is an autonomous research system utilizing frontier LLMs to automate experimental cycles across diverse domains, achieving significant performance improvements without human intervention.

Contribution

It introduces a fully autonomous, adaptable pipeline that leverages frontier LLMs for multi-domain research, including code generation, evaluation, and large-scale experiments.

Findings

GPU kernel optimization: 4.4x faster than torch.compile on average

LLM pretraining: 22% lower validation loss than baseline

Traffic forecasting: 23-25% improvement over standard baselines

Abstract

We present AlphaLab, an autonomous research harness that leverages frontier LLM agentic capabilities to automate the full experimental cycle in quantitative, computation-intensive domains. Given only a dataset and a natural-language objective, AlphaLab proceeds through three phases without human intervention: (1) it adapts to the domain and explores the data, writing analysis code and producing a research report; (2) it constructs and adversarially validates its own evaluation framework; and (3) it runs large-scale GPU experiments via a Strategist/Worker loop, accumulating domain knowledge in a persistent playbook that functions as a form of online prompt optimization. All domain-specific behavior is factored into adapters generated by the model itself, so the same pipeline handles qualitatively different tasks without modification. We evaluate AlphaLab with two frontier LLMs (GPT-5.2 and Claude Opus 4.6) on three domains: CUDA kernel optimization, where it writes GPU kernels that run 4.4x faster than torch.compile on average (up to 91x); LLM pretraining, where the full system achieves 22% lower validation loss than a single-shot baseline using the same model; and traffic forecasting, where it beats standard baselines by 23-25% after researching and implementing published model families from the literature. The two models discover qualitatively different solutions in every domain (neither dominates uniformly), suggesting that multi-model campaigns provide complementary search coverage. We additionally report results on financial time series forecasting in the appendix, and release all code at https://brendanhogan.github.io/alphalab-paper/.