How Intrinsic Motivation Underlies Embodied Open-Ended Behavior

TL;DR

This paper explores how intrinsic motivation, independent of extrinsic rewards, can drive complex, open-ended embodied behaviors, supported by a hierarchical framework and various formal theories.

Contribution

It introduces a hierarchical framework linking intrinsic motivation to behavior and reviews formal models like empowerment and free energy, highlighting their role in generating complex behaviors.

Findings

Intrinsic motivation can organize behavior hierarchically.

Formal theories promote entropic action paths.

Intrinsic motivation models generate complex, task-agnostic behaviors.

Abstract

Although most theories posit that natural behavior can be explained as maximizing some form of extrinsic reward, often called utility, some behaviors appear to be reward independent. For instance, spontaneous motor babbling in human newborns and curiosity in little kids and other animals seem to elude a simple explanation in terms of extrinsic reward maximization. Rooted in these observations, intrinsic motivation has emerged as a potentially major driver of behavior. However, only recently have several quantitative and foundational theories of intrinsic motivation been put forward. We first provide a general framework to understand behavior as being organized hierarchically: objective--intrinsic reward, or motivation--drives, goals and extrinsic reward. We next review the main formalizations of intrinsic motivation, including empowerment, the free energy principle, information-gain maximization, and the maximum occupancy principle. These theories produce complex behavior by promoting, in various ways, entropic action-state paths. The presence of a single intrinsic motivation objective breaks infinite regress, as drives and goals act only temporarily to serve the objective. Extrinsic rewards, such as sugar or protein, are just a means to achieve the objective. Bounded cognition and embodiment impose constraints and boundary conditions for the intrinsic motivation objective. By virtue of their capability to generate complex behavior in a task-agnostic manner, theories of intrinsic motivation promise to become successful generative models of open-ended, embodied behavior.