LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation

TL;DR

This paper introduces LibEvolutionEval, a benchmark for version-specific code generation that emphasizes understanding library evolution, revealing its impact on model performance and exploring mitigation strategies like documentation retrieval.

Contribution

It presents a new benchmark and analysis for code completion involving evolving libraries, highlighting the importance of library evolution understanding and proposing mitigation methods.

Findings

Library evolution significantly affects model performance.

Retrieving version-specific documentation improves code completion.

Analysis of PyTorch and Matplotlib evolution over a year.

Abstract

Recent advancements in code completion models have primarily focused on local file contexts. However, these studies do not fully capture the complexity of real-world software development, which often requires the use of rapidly-evolving public libraries. To fill the gap, we introduce LibEvolutionEval, a detailed study requiring an understanding of library evolution to perform in-line code completion accurately. LibEvolutionEval provides a version-specific code-completion task comprised of eight libraries (torch, torchvision, scipy, pil, tqdm, pyyaml, matplotlib, and pandas) as they evolve over the year along with a detailed analysis of the evolution of two popular and well-maintained public libraries: PyTorch and Matplotlib. We evaluate popular public models and find that public library evolution significantly influences model performance. We explored mitigation methods by studying how retrieved version-specific library documentation and prompting can improve the model's capability in handling these fast-evolving packages, paving a promising future path in better handling fast-evolving libraries.