CODESYNC: Synchronizing Large Language Models with Dynamic Code Evolution at Scale

TL;DR

CODESYNC introduces a benchmark and data engine to evaluate and improve large language models' ability to adapt to rapidly evolving code, especially third-party library APIs, highlighting current limitations and future directions.

Contribution

The paper presents CODESYNC and CODESYNCBENCH, novel tools for assessing and enhancing LLMs' synchronization with dynamic code changes in real-world Python libraries.

Findings

Current LLMs struggle with real-time code updates.

Existing knowledge updating methods have limited effectiveness.

Benchmark provides a comprehensive evaluation framework.

Abstract

Large Language Models (LLMs) have exhibited exceptional performance in software engineering yet face challenges in adapting to continually evolving code knowledge, particularly regarding the frequent updates of third-party library APIs. This limitation, stemming from static pre-training datasets, often results in non-executable code or implementations with suboptimal safety and efficiency. To this end, this paper introduces CODESYNC, a data engine for identifying outdated code patterns and collecting real-time code knowledge updates from Python third-party libraries. Building upon CODESYNC, we develop CODESYNCBENCH, a comprehensive benchmark for assessing LLMs' ability to stay synchronized with code evolution, which covers real-world updates for 220 APIs from six Python libraries. Our benchmark offers 3,300 test cases across three evaluation tasks and an update-aware instruction tuning dataset consisting of 2,200 training samples. Extensive experiments on 14 state-of-the-art LLMs reveal that they struggle with dynamic code evolution, even with the support of advanced knowledge updating methods (e.g., DPO, ORPO, and SimPO). We believe that our benchmark can offer a strong foundation for the development of more effective methods for real-time code knowledge updating in the future. The experimental code and dataset are publicly available at: https://github.com/Lucky-voyage/Code-Sync.