ExpThink: Experience-Guided Reinforcement Learning for Adaptive Chain-of-Thought Compression

TL;DR

ExpThink introduces an experience-guided reinforcement learning framework that dynamically balances reasoning accuracy and token efficiency, significantly reducing response length while improving performance on mathematical reasoning tasks.

Contribution

It proposes a novel RL approach with experience-guided reward shaping and difficulty-adaptive advantage to enhance CoT compression without manual tuning.

Findings

Reduces average response length by up to 77%.

Achieves up to 3x higher accuracy-efficiency ratio.

Outperforms existing RL-based compression methods.

Abstract

Large reasoning models (LRMs) achieve strong performance via extended chain-of-thought (CoT) reasoning, yet suffer from excessive token consumption and high inference latency. Existing reinforcement learning (RL) approaches for CoT compression rely on uniform, static length penalties that neglect model capability dynamics and problem-level difficulty variation. We propose \textbf{ExpThink}\xspace, an RL framework that addresses both dimensions through two complementary mechanisms. First, \emph{experience-guided reward shaping} tracks the shortest correct solution found so far for each problem and applies a three-tier reward: full credit for concise correct responses, discounted credit for verbose correct ones, and zero for incorrect ones. The threshold tightens automatically with model improvement, forming a self-evolving curriculum that requires no manual scheduling. Second, \emph{difficulty-adaptive advantage} replaces standard deviation normalization with correct-count normalization, yielding monotonically difficulty-scaled gradients that amplify learning on hard problems to preserve accuracy while suppressing gradients on easy ones to encourage brevity. Together, these mechanisms enforce an accuracy-first, compression-second training objective. Experiments on multiple mathematical reasoning benchmarks demonstrate that \textbf{ExpThink}\xspace reduces average response length by up to 77\% while simultaneously improving accuracy, achieving up to $3\times$ higher accuracy-efficiency ratio (accuracy divided by average token count) than the vanilla baseline and outperforming existing RL-based compression methods on both metrics.