Think Dense, Not Long: Dynamic Decoupled Conditional Advantage for Efficient Reasoning

TL;DR

This paper introduces DDCA, a method that improves reasoning efficiency in reinforcement learning by dynamically adjusting penalties based on problem difficulty, reducing verbosity without sacrificing accuracy.

Contribution

The paper proposes DDCA, a novel approach that decouples efficiency from correctness by dynamically scaling penalties, addressing structural issues in RLVR for better reasoning performance.

Findings

Reduces generated tokens by ~60% on simple tasks

Maintains or improves accuracy across benchmarks

Adapts penalty strength based on problem difficulty

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) can elicit strong multi-step reasoning, yet it often encourages overly verbose traces. Moreover, naive length penalties in group-relative optimization can severely hurt accuracy. We attribute this failure to two structural issues: (i) Dilution of Length Baseline, where incorrect responses (with zero length reward) depress the group baseline and over-penalize correct solutions; and (ii) Difficulty-Penalty Mismatch, where a static penalty cannot adapt to problem difficulty, suppressing necessary reasoning on hard instances while leaving redundancy on easy ones. We propose Dynamic Decoupled Conditional Advantage (DDCA) to decouple efficiency optimization from correctness. DDCA computes length advantages conditionally within the correct-response cluster to eliminate baseline dilution, and dynamically scales the penalty strength using the group pass rate as a proxy for difficulty. Experiments on GSM8K, MATH500, AMC23, and AIME25 show that DDCA consistently improves the efficiency--accuracy trade-off relative to adaptive baselines, reducing generated tokens by approximately 60% on simpler tasks (e.g., GSM8K) versus over 20% on harder benchmarks (e.g., AIME25), thereby maintaining or improving accuracy. Code is available at https://github.com/alphadl/DDCA.