Unsupervised Control Through Non-Parametric Discriminative Rewards

TL;DR

This paper introduces an unsupervised reinforcement learning method where agents learn to achieve perceptual goals by jointly training a goal-conditioned policy and a learned reward function, without requiring expert data or handcrafted rewards.

Contribution

The paper proposes a novel unsupervised learning algorithm that jointly learns a goal-conditioned policy and a non-parametric reward function based on similarity, enabling goal achievement without supervision.

Findings

Successfully learned to reach diverse goals in Atari, DeepMind Control Suite, and DeepMind Lab.

Reward function reflects controllable aspects of environment, not just observation similarity.

Demonstrated effectiveness of unsupervised goal learning in complex environments.

Abstract

Learning to control an environment without hand-crafted rewards or expert data remains challenging and is at the frontier of reinforcement learning research. We present an unsupervised learning algorithm to train agents to achieve perceptually-specified goals using only a stream of observations and actions. Our agent simultaneously learns a goal-conditioned policy and a goal achievement reward function that measures how similar a state is to the goal state. This dual optimization leads to a co-operative game, giving rise to a learned reward function that reflects similarity in controllable aspects of the environment instead of distance in the space of observations. We demonstrate the efficacy of our agent to learn, in an unsupervised manner, to reach a diverse set of goals on three domains -- Atari, the DeepMind Control Suite and DeepMind Lab.