Distribution-Centric Policy Optimization Dominates Exploration-Exploitation Trade-off

TL;DR

This paper introduces Distribution-Centric Policy Optimization (DCPO), a novel reinforcement learning approach that controls exploration by guiding policies with target distributions, improving stability and efficiency over existing sample-centric methods.

Contribution

The paper presents the first distribution-centric perspective in RL, reformulating entropy regulation as a distribution-level regularization for better exploration control.

Findings

DCPO outperforms GRPO by about 20% on average across benchmarks.

DCPO enables controllable, on-policy entropy regulation without external samples.

Distribution-level regularization enhances exploration stability and efficiency.

Abstract

The exploration-exploitation (EE) trade-off is a central challenge in reinforcement learning (RL) for large language models (LLMs). With Group Relative Policy Optimization (GRPO), training tends to be exploitation driven: entropy decreases monotonically, samples convergence, and exploration fades. Most existing fixes are \textbf{sample-centric}: they seek or bonus rare samples, assuming exploration comes from novel trajectories and tokens. These heuristics depend on the "luck" of informative samples, lack principled control of the policy, and often yield limited or inconsistent gains. In this work, we are the first to introduce a \textbf{distribution-centric} perspective for RL, in which exploration is always guided by a "better" target distribution, and reveal that a policy's ability to resist entropy collapse is governed by the distribution itself rather than individual samples. Building on this insight, we propose Distribution-Centric Policy Optimization (DCPO), which reformulates entropy regulation as distribution-level regularization. DCPO achieves controllable entropy fully on-policy without sampling from external distributions, enabling efficient exploration while maintaining training stability. Across multiple models and seven benchmarks, DCPO improves over GRPO by about 20\% on average. Overall, DCPO replaces sample-level heuristics with distribution-level principles, offering a theoretically grounded and flexible framework for controllable exploration and a stronger EE trade-off. The code is available in https://github.com/597358816/DCPO.