SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution

TL;DR

SWE Atlas is a comprehensive benchmark suite for evaluating coding agents across software engineering tasks, emphasizing practical workflows, quality metrics, and real-world applicability.

Contribution

It introduces a novel benchmark targeting underrepresented tasks, with detailed evaluation protocols and analysis of state-of-the-art models' performance.

Findings

GPT-5.4 and Opus 4.7 outperform other models

Open-weight models perform poorly overall

Top models rely on codebase exploration and runtime reasoning

Abstract

We introduce SWE Atlas, a benchmark suite for coding agents spanning three professional software engineering workflows: Codebase Q&A (124 tasks), Test Writing (90 tasks), and Refactoring (70 tasks). SWE Atlas differs from prior SWE benchmarks in three key ways: it targets underrepresented but practically important task categories, uses comprehensive category-specific evaluation protocols, and adopts under-specified, agentic task formulations that better reflect real-world usage. Its evaluation framework combines programmatic checks with rubric-based assessment. This goes beyond functional correctness, evaluating software engineering quality, including test and refactor completeness, maintainability, reusable abstractions, and codebase hygiene. We evaluate a range of frontier and open-weight models on SWE Atlas and find that GPT-5.4 and Opus 4.7 achieve the strongest overall performance, while even the best open-weight models score poorly. Our analysis suggests that top models rely on extensive codebase exploration and runtime-driven reasoning. However, even top models consistently struggle with subtle edge cases, complex runtime analysis, and adherence to software engineering best practices. Overall, SWE Atlas provides a complementary evaluation suite for measuring both correctness and engineering quality in coding agents.