Towards Effective Experiential Learning: Dual Guidance for Utilization and Internalization

TL;DR

This paper introduces DGO, a framework that enhances reinforcement learning from verifiable rewards in large language models by effectively utilizing and internalizing external and internal experiences, leading to improved reasoning capabilities.

Contribution

DGO is a novel unified framework that constructs an experience bank and guides exploration, improving RLVR training by mimicking human-like internalization of experience.

Findings

DGO outperforms baseline methods in reasoning tasks.

Utilizing experience banks improves exploration efficiency.

Internalization of experience enhances model stability.

Abstract

Recently, reinforcement learning~(RL) has become an important approach for improving the capabilities of large language models~(LLMs). In particular, reinforcement learning from verifiable rewards~(RLVR) has emerged as a promising paradigm for reasoning tasks. However, existing RL-based training still remains only a rough approximation to human learning. Human learners leverage both external and internal experience to guide exploration and gradually internalize useful trajectories into stable knowledge. Motivated by this gap, we ask: how can LLMs better utilize and internalize experience during RLVR training? To answer this question, we propose \textbf{D}ual \textbf{G}uidance \textbf{O}ptimization~(\textbf{DGO}), a unified framework that leverages \emph{external} and \emph{internal experience} to improve training effectiveness. Specifically, DGO first constructs an experience bank from previously explored trajectories. The policy then performs exploration under the joint guidance of the experience bank and the model's internal knowledge. The resulting trajectories are further used to refine the experience bank and optimize model parameters, forming a closed loop of experience utilization and internalization. Experiments show that DGO consistently outperforms baseline methods, suggesting that better utilization and internalization of experience lead to more effective reasoning.