"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistants

TL;DR

This study analyzes extensive user reviews of AI coding assistants to understand developer perceptions, revealing key user needs, concerns, and expectations to guide future improvements in these tools.

Contribution

It provides a novel taxonomy of user feedback based on real-world reviews, highlighting specific user demands and dissatisfaction points for AI coding assistants.

Findings

Most AI coding assistants are recent, with over 90% released in the last two years.

Users value context-aware, customizable, and resource-efficient features.

The study offers practical suggestions for enhancing AI coding assistants based on user feedback.

Abstract

This paper aims to explore fundamental questions in the era when AI coding assistants like GitHub Copilot are widely adopted: what do developers truly value and criticize in AI coding assistants, and what does this reveal about their needs and expectations in real-world software development? Unlike previous studies that conduct observational research in controlled and simulated environments, we analyze extensive, first-hand user reviews of AI coding assistants, which capture developers' authentic perspectives and experiences drawn directly from their actual day-to-day work contexts. We identify 1,085 AI coding assistants from the Visual Studio Code Marketplace. Although they only account for 1.64% of all extensions, we observe a surge in these assistants: over 90% of them are released within the past two years. We then manually analyze the user reviews sampled from 32 AI coding assistants that have sufficient installations and reviews to construct a comprehensive taxonomy of user concerns and feedback about these assistants. We manually annotate each review's attitude when mentioning certain aspects of coding assistants, yielding nuanced insights into user satisfaction and dissatisfaction regarding specific features, concerns, and overall tool performance. Built on top of the findings-including how users demand not just intelligent suggestions but also context-aware, customizable, and resource-efficient interactions-we propose five practical implications and suggestions to guide the enhancement of AI coding assistants that satisfy user needs.