MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI

TL;DR

MLS-Bench is a comprehensive benchmark designed to evaluate AI systems' ability to invent generalizable and scalable machine learning methods across diverse tasks, highlighting current limitations and guiding future research.

Contribution

The paper introduces MLS-Bench, a new benchmark with 140 tasks across 12 domains to assess AI's capacity for method invention and generalization, and provides insights into current challenges.

Findings

Current agents struggle to surpass human-designed methods reliably.

Engineering tuning is easier for agents than genuine method invention.

Increasing search, compute, or context alone does not overcome the bottleneck.

Abstract

Modern AI progress has been driven by ML methods that are generalizable across settings and scalable to larger regimes. As large language models demonstrate advanced capabilities in reasoning, coding, and engineering tasks, it is increasingly important to understand whether they can discover such methods rather than only apply existing ones. We introduce MLS-Bench, a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods. MLS-Bench contains 140 tasks across 12 domains, each requiring an agent to improve one targeted component of an ML system or algorithm and demonstrate that the improvement generalizes across controlled settings and scales. We find that current agents remain far from reliably surpassing human-designed methods, and that engineering-style tuning is easier for them than genuine method invention. We further study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance, together with case studies of their behavior. Our analyses suggest that the bottleneck is not only in proposing new methods, but also in the scientific insight needed to plan, validate, and scale claims about them. More search, compute, or context alone does not remove this bottleneck. We build and maintain a community platform for cumulative and comparable iteration, and release the data and code at https://mls-bench.com.