Improving LLM Code Generation via Requirement-Aware Curriculum Reinforcement Learning

TL;DR

This paper introduces RECRL, a requirement-aware curriculum reinforcement learning framework that enhances large language model code generation by better perceiving, optimizing, and sampling programming requirements.

Contribution

RECRL automatically perceives requirement difficulty, optimizes challenging requirements, and employs adaptive sampling to improve LLM code generation performance.

Findings

RECRL improves Pass@1 by 1.23%-5.62% over baselines.

RECRL effectively perceives requirement difficulty and optimizes training data.

Experiments on five LLMs and benchmarks validate RECRL's effectiveness.

Abstract

Code generation, which aims to automatically generate source code from given programming requirements, has the potential to substantially improve software development efficiency. With the rapid advancement of large language models (LLMs), LLM-based code generation has attracted widespread attention from both academia and industry. However, as programming requirements become increasingly complex, existing LLMs still exhibit notable performance limitations. To address this challenge, recent studies have proposed training-based curriculum reinforcement learning (CRL) strategies to improve LLM code generation performance. Despite their effectiveness, existing CRL approaches suffer from several limitations, including misaligned requirement difficulty perception, the absence of requirement difficulty optimization, and suboptimal curriculum sampling strategies. In CRL-based code generation, programming requirements serve as the sole input to the model, making their quality and difficulty critical to training effectiveness. Motivated by insights from software requirements engineering, we propose RECRL, a novel requirement-aware curriculum reinforcement learning framework for enhancing LLM-based code generation. RECRL automatically perceives model-specific requirement difficulty, optimizes challenging requirements to improve training data utilization, and employs an adaptive curriculum sampling strategy to construct training batches with smoothly varying difficulty. Extensive experiments on five state-of-the-art LLMs across five widely-used code generation benchmarks by comparing with five state-of-the-art baselines, demonstrate the significant effectiveness of RECRL. For example, RECRL achieves an average Pass@1 improvement of 1.23%-5.62% over all state-of-the-art baselines.