Accelerating Learning in Constructive Predictive Frameworks with the Successor Representation

TL;DR

This paper introduces the use of the successor representation (SR) to enhance learning speed and sample efficiency in constructive predictive frameworks based on general value functions (GVFs), especially in dynamic, real-world environments.

Contribution

The paper demonstrates that SR-based predictions can significantly accelerate learning and improve sample efficiency in incremental, continual learning scenarios involving GVFs.

Findings

SR improves learning speed in grid-world experiments.

SR enhances sample efficiency on a physical robot arm.

The approach supports incremental addition of predictions in dynamic environments.

Abstract

Here we propose using the successor representation (SR) to accelerate learning in a constructive knowledge system based on general value functions (GVFs). In real-world settings like robotics for unstructured and dynamic environments, it is infeasible to model all meaningful aspects of a system and its environment by hand due to both complexity and size. Instead, robots must be capable of learning and adapting to changes in their environment and task, incrementally constructing models from their own experience. GVFs, taken from the field of reinforcement learning (RL), are a way of modeling the world as predictive questions. One approach to such models proposes a massive network of interconnected and interdependent GVFs, which are incrementally added over time. It is reasonable to expect that new, incrementally added predictions can be learned more swiftly if the learning process leverages knowledge gained from past experience. The SR provides such a means of separating the dynamics of the world from the prediction targets and thus capturing regularities that can be reused across multiple GVFs. As a primary contribution of this work, we show that using SR-based predictions can improve sample efficiency and learning speed in a continual learning setting where new predictions are incrementally added and learned over time. We analyze our approach in a grid-world and then demonstrate its potential on data from a physical robot arm.