Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization

TL;DR

Policy Split introduces a dual-mode policy framework for LLM reinforcement learning, enhancing exploration without sacrificing accuracy through collaborative entropy regularization.

Contribution

It presents a novel dual-mode policy paradigm with shared parameters and tailored entropy regularization, improving exploration in LLM RL tasks.

Findings

Outperforms existing entropy-guided RL baselines across various model sizes.

Enables dual-mode exploration with distinct behavioral patterns.

Enhances both general and creative task performance.

Abstract

To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifurcates the policy into normal and high-entropy modes with a high-entropy prompt. While sharing model parameters, the two modes undergo collaborative dual-mode entropy regularization tailored to distinct objectives. Specifically, the normal mode optimizes for task correctness, while the high-entropy mode incorporates a preference for exploration, and the two modes learn collaboratively. Extensive experiments demonstrate that our approach consistently outperforms established entropy-guided RL baselines across various model sizes in general and creative tasks. Further analysis reveals that Policy Split facilitates dual-mode exploration, where the high-entropy mode generates distinct behavioral patterns to the normal mode, providing unique learning signals.