Examining the Use and Impact of an AI Code Assistant on Developer Productivity and Experience in the Enterprise

TL;DR

This study evaluates IBM's Watsonx Code Assistant, an AI tool for developers, revealing its mixed impact on productivity and user perceptions through surveys and usability tests.

Contribution

It provides empirical insights into how an enterprise AI coding assistant affects developer productivity, expectations, and responsibility perceptions.

Findings

AI assistant often increases productivity but benefits vary among users.

Developers have diverse expectations regarding speed and quality of AI-generated code.

Ownership and responsibility for generated code are key considerations for users.

Abstract

AI assistants are being created to help software engineers conduct a variety of coding-related tasks, such as writing, documenting, and testing code. We describe the use of the watsonx Code Assistant (WCA), an LLM-powered coding assistant deployed internally within IBM. Through surveys of two user cohorts (N=669) and unmoderated usability testing (N=15), we examined developers' experiences with WCA and its impact on their productivity. We learned about their motivations for using (or not using) WCA, we examined their expectations of its speed and quality, and we identified new considerations regarding ownership of and responsibility for generated code. Our case study characterizes the impact of an LLM-powered assistant on developers' perceptions of productivity and it shows that although such tools do often provide net productivity increases, these benefits may not always be experienced by all users.