Usage, Effects and Requirements for AI Coding Assistants in the Enterprise: An Empirical Study

TL;DR

This paper investigates the adoption, impact, and requirements of AI coding assistants and code LLMs in enterprise software engineering through surveys of developers and existing user studies.

Contribution

It provides empirical insights into enterprise usage, user experience, and necessary features for AI coding assistants, filling a gap in understanding their readiness for real-world applications.

Findings

Developers see potential but have concerns about reliability and integration.

User experience varies across domains and skill levels.

Key requirements include accuracy, security, and seamless integration.

Abstract

The rise of large language models (LLMs) has accelerated the development of automated techniques and tools for supporting various software engineering tasks, e.g., program understanding, code generation, software testing, and program repair. As CodeLLMs are being employed toward automating these tasks, one question that arises, especially in enterprise settings, is whether these coding assistants and the code LLMs that power them are ready for real-world projects and enterprise use cases, and how do they impact the existing software engineering process and user experience. In this paper we survey 57 developers from different domains and with varying software engineering skill about their experience with AI coding assistants and CodeLLMs. We also reviewed 35 user surveys on the usage, experience and expectations of professionals and students using AI coding assistants and CodeLLMs. Based on our study findings and analysis of existing surveys, we discuss the requirements for AI-powered coding assistants.