From Curiosity to Competence: How World Models Interact with the Dynamics of Exploration

TL;DR

This paper explores how internal representations in agents mediate the balance between curiosity-driven exploration and competence in control, comparing model-based agents with handcrafted and learned models, revealing a co-evolution of exploration and representation.

Contribution

It introduces a formal framework connecting cognitive theories of motivation with reinforcement learning, demonstrating how internal representations influence exploration and competence trade-offs.

Findings

Handcrafted state abstractions guide exploration patterns distinctly.

Learning internal world models fosters a two-way interaction between exploration and representation.

Balancing curiosity and competence enhances exploration efficiency.

Abstract

What drives an agent to explore the world while also maintaining control over the environment? From a child at play to scientists in the lab, intelligent agents must balance curiosity (the drive to seek knowledge) with competence (the drive to master and control the environment). Bridging cognitive theories of intrinsic motivation with reinforcement learning, we ask how evolving internal representations mediate the trade-off between curiosity (novelty or information gain) and competence (empowerment). We compare two model-based agents using handcrafted state abstractions (Tabular) or learning an internal world model (Dreamer). The Tabular agent shows curiosity and competence guide exploration in distinct patterns, while prioritizing both improves exploration. The Dreamer agent reveals a two-way interaction between exploration and representation learning, mirroring the developmental co-evolution of curiosity and competence. Our findings formalize adaptive exploration as a balance between pursuing the unknown and the controllable, offering insights for cognitive theories and efficient reinforcement learning.