A Geometric Perspective on Optimal Representations for Reinforcement Learning

TL;DR

This paper introduces a geometric framework for reinforcement learning representation, emphasizing the role of adversarial value functions as auxiliary tasks to improve learning efficiency.

Contribution

It presents a novel geometric perspective on value function approximation, linking auxiliary tasks to minimizing errors on adversarial value functions, and relates this to proto-value functions.

Findings

Using AVFs as auxiliary tasks improves learning in the four-room domain.

The geometric formulation provides formal evidence for the effectiveness of value functions as auxiliary tasks.

AVFs are shown to be a natural and effective choice for auxiliary tasks in RL.

Abstract

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation to minimize the (linear) approximation of the value function of all stationary policies for a given environment. We show that this optimization reduces to making accurate predictions regarding a special class of value functions which we call adversarial value functions (AVFs). We demonstrate that using value functions as auxiliary tasks corresponds to an expected-error relaxation of our formulation, with AVFs a natural candidate, and identify a close relationship with proto-value functions (Mahadevan, 2005). We highlight characteristics of AVFs and their usefulness as auxiliary tasks in a series of experiments on the four-room domain.