Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents

TL;DR

Breakpoint is a scalable benchmarking approach that automatically creates complex code-repair tasks to evaluate large language models' system-level reasoning abilities across varying difficulty levels.

Contribution

It introduces a novel, automated method for generating diverse, scalable code-repair benchmarks that evaluate both local and system-level reasoning in LLMs.

Findings

Models' success rates decrease from 55% to 0% as task difficulty increases.

The methodology can generate an arbitrary number of tasks with controlled difficulty.

State-of-the-art models struggle with highly complex, system-level reasoning tasks.

Abstract

Benchmarks for large language models (LLMs) have predominantly assessed short-horizon, localized reasoning. Existing long-horizon suites (e.g. SWE-bench) rely on manually curated issues, so expanding or tuning difficulty demands expensive human effort and evaluations quickly saturate. However, many real-world tasks, such as software engineering or scientific research, require agents to rapidly comprehend and manipulate novel, complex structures dynamically; evaluating these capabilities requires the ability to construct large and varied sets of problems for agents to solve. We introduce Breakpoint, a benchmarking methodology that automatically generates code-repair tasks by adversarially corrupting functions within real-world software repositories. Breakpoint systematically controls task difficulty along two clear dimensions: local reasoning (characterized by code complexity metrics such as cyclomatic complexity) and system-level reasoning (characterized by call-graph centrality and the number of simultaneously corrupted interdependent functions). In experiments across more than 900 generated tasks we demonstrate that our methodology can scale to arbitrary difficulty, with state-of-the-art models' success rates ranging from 55% on the easiest tasks down to 0% on the hardest.