Upside Down Reinforcement Learning with Policy Generators

TL;DR

This paper introduces UDRLPG, a novel method that uses hypernetworks to generate policies conditioned on commands, improving sample efficiency and enabling zero-shot generalization in reinforcement learning tasks.

Contribution

The paper extends Upside Down Reinforcement Learning by integrating hypernetworks to generate command-conditioned policies without requiring an evaluator, enhancing efficiency and generalization.

Findings

Achieves competitive performance and high returns in RL tasks.

Can generalize to unseen returns zero-shot.

Improves empirical convergence despite increased variance.

Abstract

Upside Down Reinforcement Learning (UDRL) is a promising framework for solving reinforcement learning problems which focuses on learning command-conditioned policies. In this work, we extend UDRL to the task of learning a command-conditioned generator of deep neural network policies. We accomplish this using Hypernetworks - a variant of Fast Weight Programmers, which learn to decode input commands representing a desired expected return into command-specific weight matrices. Our method, dubbed Upside Down Reinforcement Learning with Policy Generators (UDRLPG), streamlines comparable techniques by removing the need for an evaluator or critic to update the weights of the generator. To counteract the increased variance in last returns caused by not having an evaluator, we decouple the sampling probability of the buffer from the absolute number of policies in it, which, together with a simple weighting strategy, improves the empirical convergence of the algorithm. Compared with existing algorithms, UDRLPG achieves competitive performance and high returns, sometimes outperforming more complex architectures. Our experiments show that a trained generator can generalize to create policies that achieve unseen returns zero-shot. The proposed method appears to be effective in mitigating some of the challenges associated with learning highly multimodal functions. Altogether, we believe that UDRLPG represents a promising step forward in achieving greater empirical sample efficiency in RL. A full implementation of UDRLPG is publicly available at https://github.com/JacopoD/udrlpg_