Successor Feature Representations

TL;DR

This paper introduces Successor Feature Representations (SFR), a new method in reinforcement learning that enables transfer learning for general reward functions, overcoming limitations of previous successor feature approaches.

Contribution

The paper proposes SFR, a novel successor representation formulation that supports transfer for arbitrary reward functions and provides convergence guarantees.

Findings

SFR outperforms SF in transfer tasks with general reward functions.

SFR demonstrates convergence and transfer guarantees.

Experimental results show SFR's advantage over existing methods.

Abstract

Transfer in Reinforcement Learning aims to improve learning performance on target tasks using knowledge from experienced source tasks. Successor Representations (SR) and their extension Successor Features (SF) are prominent transfer mechanisms in domains where reward functions change between tasks. They reevaluate the expected return of previously learned policies in a new target task to transfer their knowledge. The SF framework extended SR by linearly decomposing rewards into successor features and a reward weight vector allowing their application in high-dimensional tasks. But this came with the cost of having a linear relationship between reward functions and successor features, limiting its application to tasks where such a linear relationship exists. We propose a novel formulation of SR based on learning the cumulative discounted probability of successor features, called Successor Feature Representations (SFR). Crucially, SFR allows to reevaluate the expected return of policies for general reward functions. We introduce different SFR variations, prove its convergence, and provide a guarantee on its transfer performance. Experimental evaluations based on SFR with function approximation demonstrate its advantage over SF not only for general reward functions, but also in the case of linearly decomposable reward functions.