Shorten After You're Right: Lazy Length Penalties for Reasoning RL

TL;DR

This paper introduces three reward-based techniques integrated into reinforcement learning to effectively shorten reasoning paths in large models, reducing response length significantly without extra training stages.

Contribution

The paper proposes novel reward designs for RL that directly reduce reasoning path length in large models without additional training stages.

Findings

40% reduction in response length in logic reasoning tasks

33% reduction in response length in math problems

Performance maintained or improved despite shorter responses

Abstract

Large reasoning models, such as OpenAI o1 or DeepSeek R1, have demonstrated remarkable performance on reasoning tasks but often incur a long reasoning path with significant memory and time costs. Existing methods primarily aim to shorten reasoning paths by introducing additional training data and stages. In this paper, we propose three critical reward designs integrated directly into the reinforcement learning process of large reasoning models, which reduce the response length without extra training stages. Experiments on four settings show that our method significantly decreases response length while maintaining or even improving performance. Specifically, in a logic reasoning setting, we achieve a 40% reduction in response length averaged by steps alongside a 14% gain in performance. For math problems, we reduce response length averaged by steps by 33% while preserving performance.