Between Rate-Distortion Theory & Value Equivalence in Model-Based Reinforcement Learning

TL;DR

This paper explores how rate-distortion theory can be applied to approximate value equivalence in model-based reinforcement learning, enabling near-optimal decision-making with simplified environment models under capacity constraints.

Contribution

It introduces an algorithm that synthesizes approximate environment models using rate-distortion principles, addressing the challenge of intractable environment complexity and limited agent capacity.

Findings

Proposes a formal framework for approximate value equivalence.

Demonstrates how rate-distortion theory guides environment model compression.

Shows potential for near-optimal policies with simplified models.

Abstract

The quintessential model-based reinforcement-learning agent iteratively refines its estimates or prior beliefs about the true underlying model of the environment. Recent empirical successes in model-based reinforcement learning with function approximation, however, eschew the true model in favor of a surrogate that, while ignoring various facets of the environment, still facilitates effective planning over behaviors. Recently formalized as the value equivalence principle, this algorithmic technique is perhaps unavoidable as real-world reinforcement learning demands consideration of a simple, computationally-bounded agent interacting with an overwhelmingly complex environment. In this work, we entertain an extreme scenario wherein some combination of immense environment complexity and limited agent capacity entirely precludes identifying an exactly value-equivalent model. In light of this, we embrace a notion of approximate value equivalence and introduce an algorithm for incrementally synthesizing simple and useful approximations of the environment from which an agent might still recover near-optimal behavior. Crucially, we recognize the information-theoretic nature of this lossy environment compression problem and use the appropriate tools of rate-distortion theory to make mathematically precise how value equivalence can lend tractability to otherwise intractable sequential decision-making problems.