CoScale-RL: Efficient Post-Training by Co-Scaling Data and Computation

TL;DR

CoScale-RL introduces a novel scaling strategy that enhances data and computational efficiency in training large reasoning models, significantly improving accuracy without extensive datasets.

Contribution

The paper proposes CoScale-RL, a new method for post-training scaling that improves model performance by collecting multiple solutions and using re-distillation, without large additional datasets.

Findings

Achieves an average 3.76× accuracy improvement on four benchmarks.

Effectively stabilizes reinforcement learning through scaled rollout computation.

Enhances reasoning ability boundary without extensive supervised fine-tuning datasets.

Abstract

Training Large Reasoning Model (LRM) is usually unstable and unpredictable, especially on hard problems or weak foundation models. We found that the current post-training scaling strategy can still improve on these cases. We propose CoScale-RL, a novel scaling strategy with better data and computational efficiency. We first scale up solutions to make problems solvable. The core idea is to collect multiple solutions for each problem, rather than simply enlarging the dataset. Then, we scale up rollout computation to stabilize Reinforcement Learning. We further leverage a model merge technique called Re-distillation to sustain or even improve computational efficiency when scaling up. Our method significantly improves data and computational efficiency, with an average 3.76$\times$ accuracy improvement on four benchmarks. CoScale-RL is able to improve an LRM's ability boundary without an extensive SFT dataset. Our method provides a new scaling direction to further improve LRM's reasoning ability.