Trust Calibration in IDEs: Paving the Way for Widespread Adoption of AI Refactoring

TL;DR

This paper discusses the importance of trust calibration in AI-assisted code refactoring within IDEs, emphasizing safeguards and user interaction to promote widespread adoption of LLM-based tools.

Contribution

It proposes integrating trustworthy safeguards and trust-building interactions in IDEs for AI refactoring, based on human factors research and industry collaboration.

Findings

Development of novel LLM safeguards

Design of user interactions conveying trust

Large-scale repository analysis and A/B testing

Abstract

In the software industry, the drive to add new features often overshadows the need to improve existing code. Large Language Models (LLMs) offer a new approach to improving codebases at an unprecedented scale through AI-assisted refactoring. However, LLMs come with inherent risks such as braking changes and the introduction of security vulnerabilities. We advocate for encapsulating the interaction with the models in IDEs and validating refactoring attempts using trustworthy safeguards. However, equally important for the uptake of AI refactoring is research on trust development. In this position paper, we position our future work based on established models from research on human factors in automation. We outline action research within CodeScene on development of 1) novel LLM safeguards and 2) user interaction that conveys an appropriate level of trust. The industry collaboration enables large-scale repository analysis and A/B testing to continuously guide the design of our research interventions.