Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories

TL;DR

This empirical study analyzes the behaviour of state-of-the-art code agents during software issue resolution, revealing key strategies, differences between success and failure trajectories, and implications for building more robust autonomous systems.

Contribution

The paper provides the first detailed empirical analysis of code agent trajectories, uncovering behavioral patterns and factors influencing success and failure in automated software engineering.

Findings

Successful trajectories often use defensive programming and context gathering.

Failed trajectories are longer and more variable than successful ones.

Most trajectories correctly identify problematic files even in failures.

Abstract

The increasing deployment of Large Language Model (LLM) agents for complex software engineering tasks has created a need to understand their problem-solving behaviours beyond simple success metrics. While these agents demonstrate impressive capabilities in automated issue resolution, their decision-making processes remain largely opaque. This paper presents an empirical study of agent trajectories, namely the execution traces capturing the steps agents take when attempting to resolve software issues. We analyse trajectories from three state-of-the-art code agents (OpenHands, SWE-agent, and Prometheus) on the SWE-Bench benchmark, examining both successful and failed attempts. Our investigation reveals several key insights into agent behaviour. First, we identify how distinct problem-solving strategies, such as defensive programming and context gathering, enable success in different scenarios. Second, we find that failed trajectories are consistently longer and exhibit higher variance than successful ones, with failure patterns differing significantly between agents. Third, our fault localisation analysis shows that while most trajectories correctly identify problematic files (72-81\% even in failures), success depends more on achieving approximate rather than exact code modifications. These and other findings unveiled by our study, provide a foundation for understanding agent behaviour through trajectory analysis, contributing to the development of more robust and interpretable autonomous software engineering systems.