An Empirical Study on the Usage of Transformer Models for Code Completion

TL;DR

This study evaluates Transformer models like RoBERTa and T5 for code completion across various levels, revealing T5's effectiveness especially in predicting larger code segments, thus advancing understanding of their capabilities beyond token prediction.

Contribution

It provides a large-scale empirical analysis of Transformer models for code completion at multiple granularity levels, including entire statements and blocks, which was less explored before.

Findings

T5 achieves up to 69% accuracy in token prediction.

Transformer models can effectively support code completion at different granularities.

Perfect predictions reach around 29% for entire code blocks.

Abstract

Code completion aims at speeding up code writing by predicting the next code token(s) the developer is likely to write. Works in this field focused on improving the accuracy of the generated predictions, with substantial leaps forward made possible by deep learning (DL) models. However, code completion techniques are mostly evaluated in the scenario of predicting the next token to type, with few exceptions pushing the boundaries to the prediction of an entire code statement. Thus, little is known about the performance of state-of-the-art code completion approaches in more challenging scenarios in which, for example, an entire code block must be generated. We present a large-scale study exploring the capabilities of state-of-the-art Transformer-based models in supporting code completion at different granularity levels, including single tokens, one or multiple entire statements, up to entire code blocks (e.g., the iterated block of a for loop). We experimented with several variants of two recently proposed Transformer-based models, namely RoBERTa and the Text-To-Text Transfer Transformer (T5), for the task of code completion. The achieved results show that Transformer-based models, and in particular the T5, represent a viable solution for code completion, with perfect predictions ranging from ~29%, obtained when asking the model to guess entire blocks, up to ~69%, reached in the simpler scenario of few tokens masked from the same code statement.