Predictive Representations for Skill Transfer in Reinforcement Learning

TL;DR

This paper introduces Outcome-Predictive State Representations (OPSRs), a new formalism for transfer in reinforcement learning that uses state abstraction to enable skill reuse and faster learning in new tasks.

Contribution

The paper develops OPSRs as task-independent, outcome-based state abstractions and introduces OPSR-based skills to improve transfer and learning efficiency in RL.

Findings

OPSR-based skills can be learned from demonstrations.

OPSRs enable faster learning in unseen tasks.

The framework improves transfer in reinforcement learning.

Abstract

A key challenge in scaling up Reinforcement Learning is generalizing learned behaviour. Without the ability to carry forward acquired knowledge an agent is doomed to learn each task from scratch. In this paper we develop a new formalism for transfer by virtue of state abstraction. Based on task-independent, compact observations (outcomes) of the environment, we introduce Outcome-Predictive State Representations (OPSRs), agent-centered and task-independent abstractions that are made up of predictions of outcomes. We show formally and empirically that they have the potential for optimal but limited transfer, then overcome this trade-off by introducing OPSR-based skills, i.e. abstract actions (based on options) that can be reused between tasks as a result of state abstraction. In a series of empirical studies, we learn OPSR-based skills from demonstrations and show how they speed up learning considerably in entirely new and unseen tasks without any pre-processing. We believe that the framework introduced in this work is a promising step towards transfer in RL in general, and towards transfer through combining state and action abstraction specifically.