Navigate the Unknown: Enhancing LLM Reasoning with Intrinsic Motivation Guided Exploration

TL;DR

This paper introduces IMAGINE, a reinforcement learning method that enhances large language model reasoning by providing dense, exploration-driven rewards, leading to significant performance improvements on complex tasks.

Contribution

IMAGINE offers novel reward mechanisms and integration techniques to improve exploration and reasoning in large language models, addressing limitations of sparse reward signals.

Findings

Improves reasoning performance by 22.23% on AIME 2024.

Provides dense, exploration-focused rewards for better training.

Enhances exploration efficiency on difficult samples.

Abstract

Reinforcement Learning (RL) has become a key approach for enhancing the reasoning capabilities of large language models. However, prevalent RL approaches like proximal policy optimization and group relative policy optimization suffer from sparse, outcome-based rewards and weak exploration incentives, limiting their effectiveness. Specifically, sparse rewards offer limited feedback, especially on difficult problems, and introduce biases favoring familiar trajectories over novel reasoning paths. These issues critically undermine performance on complex tasks that inherently require iterative reasoning. To overcome these challenges, we propose Intrinsic MotivAtion Guided exploratIoN for Enhanced reasoning (IMAGINE), which delivers dense rewards and encourages exploration. IMAGINE introduces three innovations: a trajectory-aware exploration reward that reduces token-level bias efficiently; an error-conditioned reward allocation that promotes efficient exploration on hard samples while stabilizing training; and an advantage-preserving integration mechanism that retains distributional integrity during learning. Experiments on four public datasets show that IMAGINE improves performance by 22.23% on AIME 2024.