Self-Consistent Models and Values

TL;DR

This paper explores a novel approach in model-based reinforcement learning by encouraging learned models and value functions to be self-consistent, which improves policy evaluation and control.

Contribution

It introduces multiple self-consistency updates for model-based RL, demonstrating their effectiveness in both tabular and function approximation settings.

Findings

Self-consistency updates improve policy evaluation.

Self-consistency enhances control performance.

Appropriate self-consistency choices are crucial for benefits.

Abstract

Learned models of the environment provide reinforcement learning (RL) agents with flexible ways of making predictions about the environment. In particular, models enable planning, i.e. using more computation to improve value functions or policies, without requiring additional environment interactions. In this work, we investigate a way of augmenting model-based RL, by additionally encouraging a learned model and value function to be jointly \emph{self-consistent}. Our approach differs from classic planning methods such as Dyna, which only update values to be consistent with the model. We propose multiple self-consistency updates, evaluate these in both tabular and function approximation settings, and find that, with appropriate choices, self-consistency helps both policy evaluation and control.