Goal-Conditioned Generators of Deep Policies

TL;DR

This paper introduces goal-conditioned neural network generators that produce deep policies as weight matrices, enabling flexible, goal-specific policy creation with competitive results in continuous control tasks.

Contribution

It presents a novel method for generating deep neural network policies conditioned on goals, combining hypernetworks and policy embeddings for scalable, goal-specific policy generation.

Findings

Single policy generator achieves multiple return levels.

Method scales to deep neural networks.

Competitive performance on continuous control tasks.

Abstract

Goal-conditioned Reinforcement Learning (RL) aims at learning optimal policies, given goals encoded in special command inputs. Here we study goal-conditioned neural nets (NNs) that learn to generate deep NN policies in form of context-specific weight matrices, similar to Fast Weight Programmers and other methods from the 1990s. Using context commands of the form "generate a policy that achieves a desired expected return," our NN generators combine powerful exploration of parameter space with generalization across commands to iteratively find better and better policies. A form of weight-sharing HyperNetworks and policy embeddings scales our method to generate deep NNs. Experiments show how a single learned policy generator can produce policies that achieve any return seen during training. Finally, we evaluate our algorithm on a set of continuous control tasks where it exhibits competitive performance. Our code is public.