Learning Relative Return Policies With Upside-Down Reinforcement Learning

TL;DR

This paper explores upside-down reinforcement learning for command-conditioned policies, demonstrating its effectiveness in online learning scenarios and complex environments, thus expanding its practical applicability.

Contribution

It shows that upside-down reinforcement learning can handle commands specifying relationships between scalar values and returns, even with non-linear function approximation.

Findings

Successful online learning in tabular bandit setting

Effective control in CartPole with non-linear approximation

Potential for practical use with complex command structures

Abstract

Lately, there has been a resurgence of interest in using supervised learning to solve reinforcement learning problems. Recent work in this area has largely focused on learning command-conditioned policies. We investigate the potential of one such method -- upside-down reinforcement learning -- to work with commands that specify a desired relationship between some scalar value and the observed return. We show that upside-down reinforcement learning can learn to carry out such commands online in a tabular bandit setting and in CartPole with non-linear function approximation. By doing so, we demonstrate the power of this family of methods and open the way for their practical use under more complicated command structures.