Learning in Markov Decision Processes with Exogenous Dynamics

TL;DR

This paper investigates a structured class of Markov Decision Processes with exogenous dynamics, demonstrating improved learning guarantees and sample efficiency by exploiting the independence of exogenous components, supported by theoretical bounds and empirical validation.

Contribution

It introduces a new framework for MDPs with exogenous state components, providing tighter regret bounds and demonstrating optimality and practical benefits over standard methods.

Findings

Significantly improved regret bounds leveraging exogenous structure

Matching lower bounds confirming optimality of the dependence

Empirical results show substantial sample efficiency gains

Abstract

Reinforcement learning algorithms are typically designed for generic Markov Decision Processes (MDPs), where any state-action pair can lead to an arbitrary transition distribution. In many practical systems, however, only a subset of the state variables is directly influenced by the agent's actions, while the remaining components evolve according to exogenous dynamics and account for most of the stochasticity. In this work, we study a structured class of MDPs characterized by exogenous state components whose transitions are independent of the agent's actions. We show that exploiting this structure yields significantly improved learning guarantees, with only the size of the exogenous state space appearing in the leading terms of the regret bounds. We further establish a matching lower bound, showing that this dependence is information-theoretically optimal. Finally, we empirically validate our approach across classical toy settings and real-world-inspired environments, demonstrating substantial gains in sample efficiency compared to standard reinforcement learning methods.