Combining Behaviors with the Successor Features Keyboard

TL;DR

This paper introduces the Successor Features Keyboard (SFK), a method that enables transfer learning in complex environments by automatically discovering state-features and task encodings, improving transfer speed and applicability.

Contribution

The paper proposes SFK and CSFA, novel algorithms for transfer learning that automatically discover representations and demonstrate transfer in a challenging 3D environment.

Findings

CSFA outperforms other SF approximation methods at scale.

SFK transfers most quickly to long-horizon tasks.

First demonstration of SF transfer with discovered representations in 3D environment.

Abstract

The Option Keyboard (OK) was recently proposed as a method for transferring behavioral knowledge across tasks. OK transfers knowledge by adaptively combining subsets of known behaviors using Successor Features (SFs) and Generalized Policy Improvement (GPI). However, it relies on hand-designed state-features and task encodings which are cumbersome to design for every new environment. In this work, we propose the "Successor Features Keyboard" (SFK), which enables transfer with discovered state-features and task encodings. To enable discovery, we propose the "Categorical Successor Feature Approximator" (CSFA), a novel learning algorithm for estimating SFs while jointly discovering state-features and task encodings. With SFK and CSFA, we achieve the first demonstration of transfer with SFs in a challenging 3D environment where all the necessary representations are discovered. We first compare CSFA against other methods for approximating SFs and show that only CSFA discovers representations compatible with SF&GPI at this scale. We then compare SFK against transfer learning baselines and show that it transfers most quickly to long-horizon tasks.