Enhancing Efficiency and Exploration in Reinforcement Learning for LLMs

TL;DR

This paper proposes a dynamic, difficulty-aware reinforcement learning method for LLMs that improves training efficiency and maintains exploration by adaptively allocating rollouts and adjusting temperature.

Contribution

It introduces a novel mechanism for dynamic rollout allocation and adaptive temperature adjustment to enhance RL training of LLMs.

Findings

Improved training efficiency through difficulty-based rollout allocation.

Maintained exploration with adaptive temperature control.

Enhanced response precision without sacrificing exploratory ability.

Abstract

Reasoning large language models (LLMs) excel in complex tasks, which has drawn significant attention to reinforcement learning (RL) for LLMs. However, existing approaches allocate an equal number of rollouts to all questions during the RL process, which is inefficient. This inefficiency stems from the fact that training on simple questions yields limited gains, whereas more rollouts are needed for challenging questions to sample correct answers. Furthermore, while RL improves response precision, it limits the model's exploration ability, potentially resulting in a performance cap below that of the base model prior to RL. To address these issues, we propose a mechanism for dynamically allocating rollout budgets based on the difficulty of the problems, enabling more efficient RL training. Additionally, we introduce an adaptive dynamic temperature adjustment strategy to maintain the entropy at a stable level, thereby encouraging sufficient exploration. This enables LLMs to improve response precision while preserving their exploratory ability to uncover potential correct pathways. The code and data is available on: https://github.com/LiaoMengqi/E3-RL4LLMs