Understanding Software Engineering Agents Through the Lens of Traceability: An Empirical Study

TL;DR

This empirical study analyzes the decision-making workflows of software engineering agents using execution traces, proposing a taxonomy and examining core components like bug localization, patch generation, and test creation to improve agent effectiveness.

Contribution

It introduces the first taxonomy of SWE agent decision pathways, studies core components in depth, and compares agent-generated and human patches through large-scale analysis.

Findings

Identified three core components: bug localization, patch generation, test creation.

Analyzed strategies for successful test generation and patch production.

Compared agent and developer patches, revealing structural and stylistic differences.

Abstract

With the advent of large language models (LLMs), software engineering agents (SWE agents) have emerged as a powerful paradigm for automating a range of software tasks -- from code generation and repair to test case synthesis. These agents operate autonomously by interpreting user input and responding to environmental feedback. While various agent architectures have demonstrated strong empirical performance, the internal decision-making worfklows that drive their behavior remain poorly understood. Deeper insight into these workflows hold promise for improving both agent reliability and efficiency. In this work, we present the first systematic study of SWE agent behavior through the lens of execution traces. Our contributions are as follows: (1) we propose the first taxonomy of decision-making pathways across five representative agents; (2) using this taxonomy, we identify three core components essential to agent success -- bug localization, patch generation, and reproduction test generation -- and study each in depth; (3) we study the impact of test generation on successful patch production; and analyze strategies that can lead to successful test generation; (4) we further conduct the first large-scale code clone analysis comparing agent-generated and developer-written patches and provide a qualitative study revealing structural and stylistic differences in patch content. Together, these findings offer novel insights into agent design and open avenues for building agents that are both more effective and more aligned with human development practices.