STALL+: Boosting LLM-based Repository-level Code Completion with Static Analysis

TL;DR

This paper introduces STALL+, a framework that enhances repository-level code completion by integrating static analysis strategies with large language models, demonstrating improved effectiveness and efficiency across different programming languages and phases.

Contribution

The paper presents the first systematic study of static analysis integration in LLM-based code completion, along with an extendable framework and extensive experimental evaluation.

Findings

File-level dependency integration in prompting is most effective.

Different static analysis strategies yield varying improvements for Java and Python.

Static analysis complements retrieval-augmented generation, enhancing cost-effectiveness.

Abstract

Repository-level code completion is challenging as it involves complicated contexts from multiple files in the repository. To date, researchers have proposed two technical categories to enhance LLM-based repository-level code completion, i.e., retrieval-augmented generation (RAG) and static analysis integration. This work performs the first study on the static analysis integration in LLM-based repository-level code completion by investigating both the effectiveness and efficiency of static analysis integration strategies across different phases of code completion. We first implement a framework STALL+, which supports an extendable and customizable integration of multiple static analysis strategies into the complete pipeline of LLM-based repository-level code completion; and based on STALL+, we perform extensive experiments by including different code LLMs on the latest repository-level code completion benchmark CrossCodeEval. Our findings show that integrating file-level dependencies in prompting phase performs the best while the integration in post-processing phase performs the worse. Additionally, we observe different improvements from static analysis between dynamic languages and static languages, i.e., the best combination is prompting-phase with decoding-phase integration for Java while the best combination is prompting-phase with post-processing-phase integration for Python given the limitations of statically analyzing dynamic languages. Additionally, we find the complementarity between RAG and static analysis integration as well as their cost-effectiveness after combination.