The Value-Improvement Path: Towards Better Representations for Reinforcement Learning

TL;DR

This paper proposes a holistic approach to value prediction in reinforcement learning by modeling the entire value-improvement path, leading to improved representations and significantly better performance in Atari games.

Contribution

It introduces the concept of the value-improvement path and demonstrates how learning this path holistically enhances representation learning in RL.

Findings

Augmented RL agent with value-improvement path auxiliary task doubles performance on Atari games.

Holistic value prediction improves the accuracy of value functions for future policy improvements.

Provides new insights into auxiliary tasks and representation learning in RL.

Abstract

In value-based reinforcement learning (RL), unlike in supervised learning, the agent faces not a single, stationary, approximation problem, but a sequence of value prediction problems. Each time the policy improves, the nature of the problem changes, shifting both the distribution of states and their values. In this paper we take a novel perspective, arguing that the value prediction problems faced by an RL agent should not be addressed in isolation, but rather as a single, holistic, prediction problem. An RL algorithm generates a sequence of policies that, at least approximately, improve towards the optimal policy. We explicitly characterize the associated sequence of value functions and call it the value-improvement path. Our main idea is to approximate the value-improvement path holistically, rather than to solely track the value function of the current policy. Specifically, we discuss the impact that this holistic view of RL has on representation learning. We demonstrate that a representation that spans the past value-improvement path will also provide an accurate value approximation for future policy improvements. We use this insight to better understand existing approaches to auxiliary tasks and to propose new ones. To test our hypothesis empirically, we augmented a standard deep RL agent with an auxiliary task of learning the value-improvement path. In a study of Atari 2600 games, the augmented agent achieved approximately double the mean and median performance of the baseline agent.