Training Large Reasoning Models Efficiently via Progressive Thought Encoding

TL;DR

This paper introduces Progressive Thought Encoding, a memory-efficient fine-tuning method that enhances reasoning accuracy of large models under fixed-size caches, significantly reducing training memory requirements.

Contribution

We propose a novel Progressive Thought Encoding technique that enables large reasoning models to reason effectively with fixed-size caches, improving training efficiency and scalability.

Findings

Achieves up to 23.4% accuracy improvement on AIME benchmarks.

Outperforms LoRA-based fine-tuning by 19.3%.

Reduces memory usage during training while maintaining inference performance.

Abstract

Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency: reinforcement learning (RL) training requires long rollouts for outcome-based rewards, where autoregressive decoding dominates time and memory usage. While sliding-window cache strategies can bound memory, they disrupt long-context reasoning and degrade performance. We introduce Progressive Thought Encoding, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches. By progressively encoding intermediate reasoning into fixed-size vector representations, our approach eliminates the need to backpropagate through full-cache rollouts, thereby reducing memory usage, while maintaining constant memory during inference. Experiments on three models, including Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and DeepSeek-R1-Distill-Llama-8B, on six widely used challenging mathematical benchmarks show consistent gains: our method achieves +19.3% improvement over LoRA-based fine-tuning and +29.9% over LRMs without fine-tuning on average, with up to +23.4 accuracy improvement on AIME2024/2025 under the same tight cache budgets. These results demonstrate that Progressive Thought Encoding not only improves reasoning accuracy but also makes RL training of LRMs substantially more efficient and scalable under real-world memory constraints.