ReCode: Updating Code API Knowledge with Reinforcement Learning

TL;DR

ReCode introduces a reinforcement learning framework that enables large language models to adapt to API updates, significantly improving their code generation accuracy in dynamic environments without degrading general coding abilities.

Contribution

The paper presents ReCode, a novel reinforcement learning approach that helps LLMs adapt to API changes, outperforming supervised fine-tuning in dynamic code generation tasks.

Findings

ReCode improves code generation performance on unseen API update tasks.

ReCode maintains LLMs' general coding abilities better than supervised fine-tuning.

Qwen2.5-Coder-7B surpasses larger models after applying ReCode.

Abstract

Large Language Models (LLMs) exhibit remarkable code generation capabilities but falter when adapting to frequent updates in external library APIs. This critical limitation, stemming from reliance on outdated API knowledge from their training data, even with access to current documentation, impedes reliable code generation in dynamic environments. To tackle this issue, we propose ReCode (rule-based Reinforcement learning for Code Update), a novel framework that mimics human programmer adaptation to API changes. Specifically, we construct a dataset of approximately 2,000 data entries to train the LLMs to perform version migration based on updated information. Then, we introduce a modified string similarity metric for code evaluation as the reward for reinforcement learning. Our experiments demonstrate that ReCode substantially boosts LLMs' code generation performance in dynamic API scenarios, especially on the unseen CodeUpdateArena task. Crucially, compared to supervised fine-tuning, ReCode has less impact on LLMs' general code generation abilities. We apply ReCode on various LLMs and reinforcement learning algorithms (GRPO and DAPO), all achieving consistent improvements. Notably, after training, Qwen2.5-Coder-7B outperforms that of the 32B parameter code instruction-tuned model and the reasoning model with the same architecture. Code is available at https://github.com/zjunlp/ReCode.