Control Models for In-IDE Code Completion

TL;DR

This paper presents control models that improve LLM-powered code completion in IDEs by filtering suggestions, reducing requests, and enhancing efficiency based on offline and online evaluations across multiple programming languages.

Contribution

Introduction of boosting- and transformer-based control models for in-IDE code completion that optimize suggestion relevance and request efficiency, validated through offline and live user studies.

Findings

Control models improve code suggestion relevance.

Models reduce unnecessary inference requests.

Enhanced completion efficiency and quality in production.

Abstract

We introduce control models for LLM-powered code completion in JetBrains IDEs: ML classifiers which trigger inference and filter the generated suggestions to better align them with users and reduce unnecessary requests. To this end, we evaluate boosting- and transformer-based architectures on an offline dataset of real code completions with n=98 users. We further evaluate the offline classification performance of our boosting-based approach on a range of syntactically diverse languages; and perform an A/B study in a production environment where they improve completion efficiency and quality metrics. With this study, we hope to demonstrate the potential in using auxiliary models for smarter in-IDE integration of LLM-driven features, highlight fruitful future directions, and open problems.