Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories

TL;DR

This paper provides a large-scale empirical analysis of the internal decision-making processes of LLM-based software engineering agents, revealing key behavioral patterns and failure modes to guide future improvements.

Contribution

It introduces a unified framework for analyzing agent trajectories, offering new insights into their reasoning coherence and operational dynamics in software engineering tasks.

Findings

Identifies key trajectory features like iteration counts and token usage.

Discovers behavioral motifs and anti-patterns linked to success and failure.

Provides actionable insights for improving agent prompting and failure diagnosis.

Abstract

Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.