Wasserstein Distance Maximizing Intrinsic Control

TL;DR

This paper introduces a new skill learning method that maximizes the Wasserstein distance of state visitation, leading to more diverse and far-reaching behaviors in an environment without relying on reward signals.

Contribution

It proposes a Wasserstein distance-based objective for skill learning, improving the diversity and coverage of learned policies over traditional mutual information approaches.

Findings

Wasserstein-based objective results in policies covering more distance in the environment.

The method outperforms diversity-based objectives in Atari environments.

Skills learned show increased state coverage and exploration.

Abstract

This paper deals with the problem of learning a skill-conditioned policy that acts meaningfully in the absence of a reward signal. Mutual information based objectives have shown some success in learning skills that reach a diverse set of states in this setting. These objectives include a KL-divergence term, which is maximized by visiting distinct states even if those states are not far apart in the MDP. This paper presents an approach that rewards the agent for learning skills that maximize the Wasserstein distance of their state visitation from the start state of the skill. It shows that such an objective leads to a policy that covers more distance in the MDP than diversity based objectives, and validates the results on a variety of Atari environments.