Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models

TL;DR

This paper introduces MicroCoder-GRPO, a novel reinforcement learning method with three key innovations, significantly improving code generation models' training stability, diversity, and performance, and provides new datasets and evaluation tools.

Contribution

It presents MicroCoder-GRPO, an improved training algorithm with three innovations, along with a new challenging dataset and evaluation framework, advancing the training of coding models.

Findings

Up to 17.6% performance improvement over baselines.

MicroCoder-Dataset yields 3x larger gains within 300 steps.

Evaluation accuracy improved by approximately 25%, with 40% faster execution.

Abstract

Modern code generation models exhibit longer outputs, accelerated capability growth, and changed training dynamics, rendering traditional training methodologies, algorithms, and datasets ineffective for improving their performance. To address these training bottlenecks, we propose MicroCoder-GRPO, an improved Group Relative Policy Optimization approach with three innovations: conditional truncation masking to improve long output potential while maintaining training stability, diversity-determined temperature selection to maintain and encourage output diversity, and removal of KL loss with high clipping ratios to facilitate solution diversity. MicroCoder-GRPO achieves up to 17.6% relative improvement over strong baselines on LiveCodeBench v6, with more pronounced gains under extended context evaluation. Additionally, we release MicroCoder-Dataset, a more challenging training corpus that achieves 3x larger performance gains than mainstream datasets on LiveCodeBench v6 within 300 training steps, and MicroCoder-Evaluator, a robust framework with approximately 25% improved evaluation accuracy and around 40% faster execution. Through comprehensive analysis across more than thirty controlled experiments, we reveal 34 training insights across seven main aspects, demonstrating that properly trained models can achieve competitive performance with larger counterparts.