Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning

TL;DR

This paper analyzes mutual information skill learning (MISL) in reinforcement learning, showing that benefits of Wasserstein-based methods like METRA can be explained within MISL, and proposes a simpler contrastive successor features approach.

Contribution

It demonstrates that the advantages of Wasserstein-based methods can be understood within MISL and introduces a new contrastive successor features method with fewer components.

Findings

Contrastive successor features retain METRA's performance

Connections established between skill learning, contrastive learning, and successor features

Ablation studies reveal key ingredients for effective skill learning

Abstract

Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.