Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychology

TL;DR

This paper demonstrates that large language models can exhibit latent learning through unrewarded exploration, leading to improved performance when rewards are later introduced, thus challenging reward-centric training paradigms.

Contribution

The study provides empirical evidence and theoretical explanations showing that LLMs can undergo latent learning during unrewarded exploration, enhancing their capabilities beyond reward-based reinforcement learning.

Findings

LLMs show performance improvements during unrewarded exploration phases.

Post-training with unrewarded exploration leads to higher competence than reward-only training.

Latent learning dynamics are consistent across multiple model families and tasks.

Abstract

Latent learning, classically theorized by Tolman, shows that biological agents (e.g., rats) can acquire internal representations of their environment without rewards, enabling rapid adaptation once rewards are introduced. In contrast, from a cognitive science perspective, reward learning remains overly dependent on external feedback, limiting flexibility and generalization. Although recent advances in the reasoning capabilities of large language models (LLMs), such as OpenAI-o1 and DeepSeek-R1, mark a significant breakthrough, these models still rely primarily on reward-centric reinforcement learning paradigms. Whether and how the well-established phenomenon of latent learning in psychology can inform or emerge within LLMs' training remains largely unexplored. In this work, we present novel findings from our experiments that LLMs also exhibit the latent learning dynamics. During an initial phase of unrewarded exploration, LLMs display modest performance improvements, as this phase allows LLMs to organize task-relevant knowledge without being constrained by reward-driven biases, and performance is further enhanced once rewards are introduced. LLMs post-trained under this two-stage exploration regime ultimately achieve higher competence than those post-trained with reward-based reinforcement learning throughout. Beyond these empirical observations, we also provide theoretical analyses for our experiments explaining why unrewarded exploration yields performance gains, offering a mechanistic account of these dynamics. Specifically, we conducted extensive experiments across multiple model families and diverse task domains to establish the existence of the latent learning dynamics in LLMs.