Understanding User Mental Models in AI-Driven Code Completion Tools: Insights from an Elicitation Study

TL;DR

This study explores developers' mental models of AI code completion tools through co-design workshops, revealing diverse preferences and the need for customizable, adaptive CCTs like ATHENA to improve developer interaction and efficiency.

Contribution

It provides new insights into developer mental models and introduces ATHENA, a customizable, adaptive code completion tool based on elicited user preferences.

Findings

Developers have diverse preferences for suggestion triggers and display modes.

Customization of activation timing, display, and explanations improves CCT usability.

ATHENA demonstrates the feasibility of adaptive, preference-aware code completion.

Abstract

Integrated Development Environments increasingly implement AI-powered code completion tools (CCTs), which promise to enhance developer efficiency, accuracy, and productivity. However, interaction challenges with CCTs persist, mainly due to mismatches between developers' mental models and the unpredictable behavior of AI-generated suggestions, which is an aspect underexplored in the literature. We conducted an elicitation study with 56 developers using co-design workshops to elicit their mental models when interacting with CCTs. Different important findings that might drive the interaction design with CCTs emerged. For example, developers expressed diverse preferences on when and how code suggestions should be triggered (proactive, manual, hybrid), where and how they are displayed (inline, sidebar, popup, chatbot), as well as the level of detail. It also emerged that developers need to be supported by customization of activation timing, display modality, suggestion granularity, and explanation content, to better fit the CCT to their preferences. To demonstrate the feasibility of these and the other guidelines that emerged during the study, we developed ATHENA, a proof-of-concept CCT that dynamically adapts to developers' coding preferences and environments, ensuring seamless integration into diverse workflows.