Beyond the Commit: Developer Perspectives on Productivity with AI Coding Assistants

TL;DR

This study explores how to effectively measure developer productivity with AI coding assistants, emphasizing a multifaceted, human-centered approach that considers both short-term and long-term impacts.

Contribution

It introduces a comprehensive, mixed-method framework for evaluating AI coding tools, highlighting the importance of long-term metrics like expertise and ownership.

Findings

Survey results show conflicting views on AI tool usefulness.

Interviews reveal six factors affecting productivity, including long-term aspects.

Long-term metrics such as technical expertise are crucial for assessment.

Abstract

Measuring developer productivity is a topic that has attracted attention from both academic research and industrial practice. In the age of AI coding assistants, it has become even more important for both academia and industry to understand how to measure their impact on developer productivity, and to reconsider whether earlier measures and frameworks still apply. This study analyzes the validity of different approaches to evaluating the productivity impacts of AI coding assistants by leveraging mixed-method research. At BNY Mellon, we conduct a survey with 2989 developer responses and 11 in-depth interviews. Our findings demonstrate that a multifaceted approach is needed to measure AI productivity impacts: survey results expose conflicting perspectives on AI tool usefulness, while interviews elicit six distinct factors that capture both short-term and long-term dimensions of productivity. In contrast to prior work, our factors highlight the importance of long-term metrics like technical expertise and ownership of work. We hope this work encourages future research to incorporate a broader range of human-centered factors, and supports industry in adopting more holistic approaches to evaluating developer productivity.