BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing?

TL;DR

BeyondSWE introduces a comprehensive benchmark for code agents that evaluates their ability to handle complex, real-world coding tasks beyond single-repo fixes, revealing significant performance gaps and challenges in integrating search and reasoning.

Contribution

The paper presents BeyondSWE, a new benchmark for evaluating code agents on complex, real-world tasks, and introduces SearchSWE, a framework for integrating search with coding to improve performance.

Findings

Frontier models achieve below 45% success on complex tasks.

No single model performs well across all task types.

Search augmentation yields inconsistent or negative performance gains.

Abstract

Current benchmarks for code agents primarily assess narrow, repository-specific fixes, overlooking critical real-world challenges such as cross-repository reasoning, domain-specialized problem solving, dependency-driven migration, and full-repository generation. To address this gap, we introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope - using 500 real-world instances across four distinct settings. Experimental results reveal a significant capability gap: even frontier models plateau below 45% success, and no single model performs consistently across task types. To systematically investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities. Our experiments show that search augmentation yields inconsistent gains and can in some cases degrade performance, highlighting the difficulty of emulating developer-like workflows that interleave search and reasoning during coding tasks. This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.