Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows

TL;DR

This study investigates how autonomous AI coding agents influence developer workflows, productivity, and experience, revealing benefits over copilots and highlighting challenges for broader adoption.

Contribution

First controlled study analyzing developer interactions with autonomous coding agents, comparing them to copilots and providing insights into workflow transformations.

Findings

Agents can complete tasks beyond copilots' capabilities

Use of agents reduces developer effort in task completion

Challenges include understanding agent behaviors for wider adoption

Abstract

Developers now have access to a growing array of increasingly autonomous AI tools for software development. While many studies examine copilots that provide chat assistance or code completions, evaluations of coding agents -- which can automatically write files and run code -- still rely on static benchmarks. We present the first controlled study of developer interactions with coding agents, characterizing how more autonomous AI tools affect productivity and experience. We evaluate two leading copilot and agentic coding assistants, recruiting participants who regularly use the former. Our results show agents can assist developers in ways that surpass copilots (e.g., completing tasks humans may not have accomplished) and reduce the effort required to finish tasks. Yet challenges remain for broader adoption, including ensuring users adequately understand agent behaviors. Our findings reveal how workflows shift with coding agents and how interactions differ from copilots, motivating recommendations for researchers and highlighting challenges in adopting agentic systems.