Proto Successor Measure: Representing the Behavior Space of an RL Agent

TL;DR

This paper introduces the Proto Successor Measure, a set of basis functions representing all possible behaviors of an RL agent, enabling zero-shot policy optimization for any reward function without further environment interaction.

Contribution

The authors propose a novel behavior representation basis that allows zero-shot policy computation in RL, independent of task-specific assumptions.

Findings

The basis functions can represent any visitation distribution behavior.

The method enables optimal policy retrieval for new rewards without additional environment interaction.

Experimental results demonstrate the approach's effectiveness in various environments.

Abstract

Having explored an environment, intelligent agents should be able to transfer their knowledge to most downstream tasks within that environment without additional interactions. Referred to as "zero-shot learning", this ability remains elusive for general-purpose reinforcement learning algorithms. While recent works have attempted to produce zero-shot RL agents, they make assumptions about the nature of the tasks or the structure of the MDP. We present Proto Successor Measure: the basis set for all possible behaviors of a Reinforcement Learning Agent in a dynamical system. We prove that any possible behavior (represented using visitation distributions) can be represented using an affine combination of these policy-independent basis functions. Given a reward function at test time, we simply need to find the right set of linear weights to combine these bases corresponding to the optimal policy. We derive a practical algorithm to learn these basis functions using reward-free interaction data from the environment and show that our approach can produce the optimal policy at test time for any given reward function without additional environmental interactions. Project page: https://agarwalsiddhant10.github.io/projects/psm.html.