On the Convergence of Bounded Agents

TL;DR

This paper explores new definitions of agent convergence in reinforcement learning, focusing on bounded agents and their internal state stability, providing theoretical insights into when an agent's behavior stabilizes.

Contribution

It introduces two novel, formal definitions of convergence for bounded agents and analyzes their properties and relationships, clarifying a key concept in reinforcement learning theory.

Findings

The first definition relates convergence to the minimal number of states needed for future behavior.

The second definition ties convergence to the stability of the agent's performance relative to internal state changes.

Both definitions align with standard views in typical reinforcement learning settings.

Abstract

When has an agent converged? Standard models of the reinforcement learning problem give rise to a straightforward definition of convergence: An agent converges when its behavior or performance in each environment state stops changing. However, as we shift the focus of our learning problem from the environment's state to the agent's state, the concept of an agent's convergence becomes significantly less clear. In this paper, we propose two complementary accounts of agent convergence in a framing of the reinforcement learning problem that centers around bounded agents. The first view says that a bounded agent has converged when the minimal number of states needed to describe the agent's future behavior cannot decrease. The second view says that a bounded agent has converged just when the agent's performance only changes if the agent's internal state changes. We establish basic properties of these two definitions, show that they accommodate typical views of convergence in standard settings, and prove several facts about their nature and relationship. We take these perspectives, definitions, and analysis to bring clarity to a central idea of the field.