Ergodicity in reinforcement learning

TL;DR

This paper explores how non-ergodic reward processes affect reinforcement learning, emphasizing the importance of ergodicity for meaningful long-term optimization and reviewing solutions for non-ergodic scenarios.

Contribution

It clarifies the impact of non-ergodic rewards in reinforcement learning and connects ergodic theory to existing solutions for optimizing individual trajectories.

Findings

Non-ergodic rewards can lead to misleading expected values.

Ergodic reward processes are crucial for meaningful long-term optimization.

Existing solutions address non-ergodic reward dynamics.

Abstract

In reinforcement learning, we typically aim to optimize the expected value of the sum of rewards an agent collects over a trajectory. However, if the process generating these rewards is non-ergodic, the expected value, i.e., the average over infinitely many trajectories with a given policy, is uninformative for the average over a single, but infinitely long trajectory. Thus, if we care about how the individual agent performs during deployment, the expected value is not a good optimization objective. In this paper, we discuss the impact of non-ergodic reward processes on reinforcement learning agents through an instructive example, relate the notion of ergodic reward processes to more widely used notions of ergodic Markov chains, and present existing solutions that optimize long-term performance of individual trajectories under non-ergodic reward dynamics.