Discrete State-Action Abstraction via the Successor Representation

TL;DR

This paper introduces DSAA, a method that automatically learns a sparse, discrete state-action abstraction using successor representations, improving efficiency in reinforcement learning by creating meaningful options for complex environments.

Contribution

The work presents a novel end-to-end trainable model for discrete abstraction in RL, leveraging successor representations and max-entropy regularization, and introduces an algorithm to derive temporally extended actions.

Findings

Effective in creating discrete abstractions for complex environments

Enhances exploration strategies in reinforcement learning

Improves downstream task performance

Abstract

While the difficulty of reinforcement learning problems is typically related to the complexity of their state spaces, Abstraction proposes that solutions often lie in simpler underlying latent spaces. Prior works have focused on learning either a continuous or dense abstraction, or require a human to provide one. Information-dense representations capture features irrelevant for solving tasks, and continuous spaces can struggle to represent discrete objects. In this work we automatically learn a sparse discrete abstraction of the underlying environment. We do so using a simple end-to-end trainable model based on the successor representation and max-entropy regularization. We describe an algorithm to apply our model, named Discrete State-Action Abstraction (DSAA), which computes an action abstraction in the form of temporally extended actions, i.e., Options, to transition between discrete abstract states. Empirically, we demonstrate the effects of different exploration schemes on our resulting abstraction, and show that it is efficient for solving downstream tasks.