Better Context Makes Better Code Language Models: A Case Study on Function Call Argument Completion

TL;DR

This paper demonstrates that incorporating non-local code context via program analyzers significantly improves function call argument completion in pretrained code language models, addressing limitations of local-context-only approaches.

Contribution

It introduces a new dataset with full project context and external dependencies, and explores methods to integrate program analyzer information into code completion models.

Findings

Providing access to function implementation improves argument prediction.

Using function usage data enhances model performance.

Different methods of integrating analyzer info affect results.

Abstract

Pretrained code language models have enabled great progress towards program synthesis. However, common approaches only consider in-file local context and thus miss information and constraints imposed by other parts of the codebase and its external dependencies. Existing code completion benchmarks also lack such context. To resolve these restrictions we curate a new dataset of permissively licensed Python packages that includes full projects and their dependencies and provide tools to extract non-local information with the help of program analyzers. We then focus on the task of function call argument completion which requires predicting the arguments to function calls. We show that existing code completion models do not yield good results on our completion task. To better solve this task, we query a program analyzer for information relevant to a given function call, and consider ways to provide the analyzer results to different code completion models during inference and training. Our experiments show that providing access to the function implementation and function usages greatly improves the argument completion performance. Our ablation study provides further insights on how different types of information available from the program analyzer and different ways of incorporating the information affect the model performance.