Why AI Agents Still Need You: Findings from Developer-Agent Collaborations in the Wild

TL;DR

This study investigates how developers collaborate with AI-powered software engineering agents in real-world settings, highlighting the importance of active engagement and iterative interaction for successful problem-solving.

Contribution

It provides empirical insights into developer-agent collaboration dynamics and identifies barriers to effective teamwork in real-world software development.

Findings

Active collaboration and iteration lead to higher success rates.

Trust issues hinder effective developer-agent interactions.

Incremental problem-solving improves resolution success.

Abstract

Software Engineering Agents (SWE agents) can autonomously perform development tasks on benchmarks like SWE Bench, but still face challenges when tackling complex and ambiguous real-world tasks. Consequently, SWE agents are often designed to allow interactivity with developers, enabling collaborative problem-solving. To understand how developers collaborate with SWE agents and the barriers they face in such interactions, we observed 19 developers using an in-IDE agent to resolve 33 open issues in repositories to which they had previously contributed. Participants successfully resolved about half of these issues, with those solving issues incrementally having greater success than those using a one-shot approach. Participants who actively collaborated with the agent and iterated on its outputs were also more successful, though they faced challenges in trusting the agent's responses and collaborating on debugging and testing. Our findings suggest that to facilitate successful collaborations, both SWE agents and developers should actively contribute to tasks throughout all stages of the software development process. SWE agents can enable this by challenging and engaging in discussions with developers, rather than being conclusive or sycophantic.