FormulaCode: Evaluating Agentic Optimization on Large Codebases

TL;DR

FormulaCode introduces a comprehensive benchmark with real-world Python codebases and multi-objective metrics to evaluate large language model agents' ability to optimize entire repositories.

Contribution

It provides the first large-scale, multi-objective benchmark for assessing agentic optimization on real-world code repositories, addressing limitations of synthetic task-based benchmarks.

Findings

LLM agents struggle with holistic, multi-objective codebase optimization.

Existing benchmarks do not adequately evaluate real-world optimization capabilities.

FormulaCode enables more realistic assessment of LLM agent performance.

Abstract

Large language model (LLM) coding agents increasingly operate at the repository level, motivating benchmarks that evaluate their ability to optimize entire codebases under realistic constraints. Existing code benchmarks largely rely on synthetic tasks, binary correctness signals, or single-objective evaluation, limiting their ability to assess holistic optimization behavior. We introduce FormulaCode, a benchmark for evaluating agentic optimization on large, real-world codebases with fine-grained, multi-objective performance metrics. FormulaCode comprises 957 performance bottlenecks mined from scientific Python repositories on GitHub, each paired with expert-authored patches and, on average, 264.6 community-maintained performance workloads per task, enabling the holistic ability of LLM agents to optimize codebases under realistic correctness and performance constraints. Our evaluations reveal that repository-scale, multi-objective optimization remains a major challenge for frontier LLM agents. Project website at: https://formula-code.github.io