APEX-SWE

TL;DR

APEX-SWE introduces a new benchmark for evaluating frontier AI models on real-world software engineering tasks, focusing on integration and observability, with the aim to assess their practical utility in software development.

Contribution

This paper presents the APEX-SWE benchmark, a novel evaluation framework for AI models on complex software engineering tasks involving integration and observability, with open-source tools and datasets.

Findings

Claude Opus 4.6 leads with 40.5% Pass@1

Performance correlates with epistemic discipline and verification practices

Open-sourced evaluation harness and dev set available

Abstract

We introduce the AI Productivity Index for Software Engineering (APEX-SWE), a benchmark for assessing whether frontier AI models can execute economically valuable software engineering work. Unlike existing evaluations that focus on narrow, well-defined tasks, APEX-SWE assesses two novel task types that reflect real-world software engineering: (1) Integration tasks (n=100), which require constructing end-to-end systems across heterogeneous cloud primitives, business applications, and infrastructure-as-code services, and (2) Observability tasks (n=100), which require debugging production failures using telemetry signals such as logs and dashboards, as well as unstructured context. We evaluated eleven frontier models for the APEX-SWE leaderboard. Claude Opus 4.6 leads the APEX-SWE leaderboard with 40.5% Pass@1, followed by Claude Opus 4.5 at 38.7%. Our analysis shows that strong performance is primarily driven by epistemic discipline, defined as the capacity to distinguish between assumptions and verified facts. It is often combined with systematic verification prior to acting. We open-source the APEX-SWE evaluation harness and a dev set (n=50).