Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR

TL;DR

This paper introduces Archer, a dual-token constraint framework for RLVR that enhances reasoning in large language models by differentiating token types during training.

Contribution

Archer is the first entropy-aware RLVR method that preserves sequence dependencies while applying distinct constraints to reasoning and knowledge tokens.

Findings

Archer outperforms strong baselines on mathematical reasoning and code generation tasks.

It improves pass@1 and pass@K performance across multiple model scales.

Differentiated token constraints lead to better reasoning and knowledge retention.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective post-training method for improving the reasoning abilities of Large Language Models (LLMs). However, existing methods mainly apply uniform optimization constraints across all tokens, ignoring their heterogeneous roles. Prior work shows that high-entropy tokens are closely tied to reasoning, while low-entropy tokens primarily encode factual knowledge, and recent approaches attempt to exploit this distinction by isolating token updates via masking or asynchronous training. We argue that such isolation breaks the sequential dependency structure of autoregressive generation, leading to suboptimal learning. To address this, we propose \textbf{Archer}, an entropy-aware RLVR framework with \textbf{dual-token constraints} that preserves joint optimization while modulating update strength across token types. Our method introduces response-level entropy normalization for stable token classification and applies differentiated clipping ranges and KL regularization to encourage exploration on reasoning tokens while preserving knowledge tokens. Experiments on mathematical reasoning and code generation benchmarks show that Archer consistently outperforms strong baselines across multiple model scales, improving both \textit{pass@1} and \textit{pass@K} performance. These results highlight the importance of respecting sequence-level dependencies when designing fine-grained RL optimization strategies for LLMs.