DiPO: Disentangled Perplexity Policy Optimization for Fine-grained Exploration-Exploitation Trade-Off

TL;DR

This paper introduces DiPO, a method that disentangles perplexity to better manage exploration and exploitation in reinforcement learning for large language models, improving their reasoning and function calling abilities.

Contribution

It proposes a novel perplexity space disentangling strategy and a bidirectional reward mechanism for fine-grained exploration-exploitation trade-off in LLM training.

Findings

Demonstrates improved performance on mathematical reasoning tasks.

Shows enhanced function calling accuracy.

Validates the effectiveness of perplexity-guided exploration.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has catalyzed significant advances in the reasoning capabilities of Large Language Models (LLMs). However, effectively managing the exploration and exploitation trade-off remains a critical challenge. In this paper, we fully analyze the exploration and exploitation dilemma of extremely hard and easy samples during the training and propose a new fine-grained trade-off mechanism. Concretely, we introduce a perplexity space disentangling strategy that divides the sample space into distinct exploration (high perplexity) and exploitation (low perplexity) subspaces, thereby mining fine-grained samples requiring exploration-exploitation trade-off. Subsequently, we propose a bidirectional reward allocation mechanism with a minimum impact on verification rewards to implement perplexity-guided exploration and exploitation, enabling more stable policy optimization. Finally, we have evaluated our method on two mainstream tasks: mathematical reasoning and function calling, and experimental results demonstrate the superiority of the proposed method, confirming its effectiveness in enhancing LLM performance by fine-grained exploration-exploitation trade-off.