A Laplacian Framework for Option Discovery in Reinforcement Learning

TL;DR

This paper introduces a Laplacian framework that leverages proto-value functions to discover options in reinforcement learning, enabling better exploration and representation without relying on environment rewards.

Contribution

It presents eigenpurposes derived from PVFs as intrinsic reward functions, revealing how options can be implicitly defined by the principal directions of the state space.

Findings

Options traverse principal directions of state space

Eigenpurposes facilitate exploration at multiple time scales

Effective in tabular and Atari domains

Abstract

Representation learning and option discovery are two of the biggest challenges in reinforcement learning (RL). Proto-value functions (PVFs) are a well-known approach for representation learning in MDPs. In this paper we address the option discovery problem by showing how PVFs implicitly define options. We do it by introducing eigenpurposes, intrinsic reward functions derived from the learned representations. The options discovered from eigenpurposes traverse the principal directions of the state space. They are useful for multiple tasks because they are discovered without taking the environment's rewards into consideration. Moreover, different options act at different time scales, making them helpful for exploration. We demonstrate features of eigenpurposes in traditional tabular domains as well as in Atari 2600 games.