On the Generalizability of Transformer Models to Code Completions of Different Lengths

TL;DR

This study investigates whether large language models for code can generalize to input lengths not seen during training, finding that current encoding schemes do not support such generalization and emphasizing the importance of training data diversity.

Contribution

The paper provides a comprehensive empirical evaluation of length generalization in encoder-decoder LLMs for code, comparing multiple encoding schemes and highlighting the need for representative training data.

Findings

None of the evaluated encoding schemes generalize well to unseen lengths.

Ensuring training data includes all relevant lengths is crucial for reliable performance.

Current solutions like Sinusoidal, xPOS, ALiBi, and T5 are insufficient for length generalization.

Abstract

The programming landscape is nowadays being reshaped by the advent of Large Language Models (LLMs) able to automate code-related tasks related to code implementation (e.g., code completion) and comprehension (e.g., code summarization). Such a paradigm shift comes with a number of implications related to how software will be written, maintained, and evolved. Also, these LLMs are extremely expensive to train, posing questions on their sustainability over time. Given their training cost, their ability to generalize, namely their ability to work on task instances different from those on which they have been trained, is an aspect worth being investigated. Previous work already showed that transformer models can successfully support code completion in a cross-project setting. However, it is unclear whether LLM are able to generalize to inputs having lengths not seen during training. For example, it is known that training a model on short instances allows to substantially reduce the training cost. However, the extent to which such a model would provide good performance on sequences having lengths not seen during training is not known. Many recent works in Natural Language Processing (NLP) tackled this problem in the context of decoder-only LLMs, i.e., xPOS and ALiBi. To assess if these solutions extend to encoder-decoder LLMs usually adopted in the code-related tasks, we present a large empirical study evaluating this generalization property of these and other encoding schemes proposed in the literature, namely Sinusoidal, xPOS, ALiBi, and T5. We found that none of these solutions successfully generalize to unseen lengths and that the only safe solution is to ensure the representativeness in the training set of all lengths likely to be encountered at inference time.