Illuminating the Three Dogmas of Reinforcement Learning under Evolutionary Light

TL;DR

This paper critically examines three foundational beliefs in reinforcement learning, proposing an evolutionary framework to revise these dogmas and explore implications for biological learning and agency.

Contribution

It introduces an evolutionary-inspired framework to reconsider core RL dogmas, linking biological evolution, agency, and multi-objective optimization.

Findings

Evolutionary dynamics can operate within individual brains.

Enrichment of the adaptation view of learning through evolutionary insights.

Analogies from evolution clarify the limits of the reward hypothesis.

Abstract

Three core tenets of reinforcement learning (RL)--concerning the definition of agency, the objective of learning, and the scope of the reward hypothesis--have been highlighted as key targets for conceptual revision, with major implications for theory and application. We propose a framework, inspired by open-ended evolutionary theory, to reconsider these three "dogmas." We revisit each assumption and address related concerns raised alongside them. To make our arguments relevant to RL as a model of biological learning, we first establish that evolutionary dynamics can plausibly operate within living brains over an individual's lifetime, and are not confined to cross-generational processes. We begin by revisiting the second dogma, drawing on evolutionary insights to enrich the "adaptation-rather-than-search" view of learning. We then address the third dogma regarding the limits of the reward hypothesis, using analogies from evolutionary fitness to illuminate the scalar reward vs. multi-objective debate. After discussing practical implications for exploration in RL, we turn to the first--and arguably most fundamental--issue: the absence of a formal account of agency. We argue that unlike the other two problems, the evolutionary paradigm alone cannot resolve the agency question, though it gestures in a productive direction. We advocate integrating ideas from origins-of-life theory, where the thermodynamics of sustenance and replication offer promising foundations for understanding agency and resource-constrained reinforcement learning in biological systems.