Compress and Control

TL;DR

This paper introduces an information-theoretic policy evaluation method that transforms compression models into value estimators for reinforcement learning, demonstrating effectiveness even with limited models on Atari games.

Contribution

It presents a novel approach linking compression models to value estimation, with theoretical guarantees and practical success in complex environments.

Findings

Consistent value estimation with powerful models under stationarity.

Effective control achieved with limited models in Atari games.

Potential scalability demonstrated for large problems.

Abstract

This paper describes a new information-theoretic policy evaluation technique for reinforcement learning. This technique converts any compression or density model into a corresponding estimate of value. Under appropriate stationarity and ergodicity conditions, we show that the use of a sufficiently powerful model gives rise to a consistent value function estimator. We also study the behavior of this technique when applied to various Atari 2600 video games, where the use of suboptimal modeling techniques is unavoidable. We consider three fundamentally different models, all too limited to perfectly model the dynamics of the system. Remarkably, we find that our technique provides sufficiently accurate value estimates for effective on-policy control. We conclude with a suggestive study highlighting the potential of our technique to scale to large problems.