Exploratory State Representation Learning

TL;DR

The paper introduces XSRL, a method that combines exploration and state representation learning to improve exploration efficiency and accelerate reinforcement learning in environments with image observations.

Contribution

XSRL jointly learns state representations and transition models while guiding exploration with a learning progress bonus, addressing exploration and SRL simultaneously.

Findings

XSRL enables efficient exploration in complex environments.

State representations learned by XSRL accelerate RL training.

The approach outperforms baseline methods in challenging environments.

Abstract

Not having access to compact and meaningful representations is known to significantly increase the complexity of reinforcement learning (RL). For this reason, it can be useful to perform state representation learning (SRL) before tackling RL tasks. However, obtaining a good state representation can only be done if a large diversity of transitions is observed, which can require a difficult exploration, especially if the environment is initially reward-free. To solve the problems of exploration and SRL in parallel, we propose a new approach called XSRL (eXploratory State Representation Learning). On one hand, it jointly learns compact state representations and a state transition estimator which is used to remove unexploitable information from the representations. On the other hand, it continuously trains an inverse model, and adds to the prediction error of this model a $k$-step learning progress bonus to form the maximization objective of a discovery policy. This results in a policy that seeks complex transitions from which the trained models can effectively learn. Our experimental results show that the approach leads to efficient exploration in challenging environments with image observations, and to state representations that significantly accelerate learning in RL tasks.