TRE: Encouraging Exploration in the Trust Region

TL;DR

This paper introduces Trust Region Entropy (TRE), a novel exploration method for reinforcement learning in large language models that mitigates tail risk issues and improves performance across various reasoning and alignment tasks.

Contribution

TRE is a new exploration technique that restricts entropy maximization within the model's trust region, addressing tail risk problems in LLMs with large vocabularies.

Findings

TRE outperforms PPO and standard entropy regularization.

Effective across mathematical reasoning, combinatorial search, and preference alignment.

Demonstrates consistent improvement in exploration quality.

Abstract

Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.