Option Transfer and SMDP Abstraction with Successor Features

TL;DR

This paper introduces a successor feature-based abstraction scheme in reinforcement learning that enables transfer of options across environments and improves planning efficiency by jointly considering state and temporal abstractions.

Contribution

We propose a novel successor feature-based abstraction method that facilitates transfer of options and state aggregation in reinforcement learning.

Findings

Effective transfer of options across different environments.

Improved planning efficiency with transferred options.

Joint state and temporal abstraction enhances generalisation.

Abstract

Abstraction plays an important role in the generalisation of knowledge and skills and is key to sample efficient learning. In this work, we study joint temporal and state abstraction in reinforcement learning, where temporally-extended actions in the form of options induce temporal abstractions, while aggregation of similar states with respect to abstract options induces state abstractions. Many existing abstraction schemes ignore the interplay of state and temporal abstraction. Consequently, the considered option policies often cannot be directly transferred to new environments due to changes in the state space and transition dynamics. To address this issue, we propose a novel abstraction scheme building on successor features. This includes an algorithm for transferring abstract options across different environments and a state abstraction mechanism that allows us to perform efficient planning with the transferred options.