Backward Curriculum Reinforcement Learning

TL;DR

This paper introduces backward curriculum reinforcement learning, which trains agents using reversed trajectories to enhance sample efficiency with minimal algorithm modifications.

Contribution

It proposes a simple yet effective backward trajectory approach that improves sample efficiency in reinforcement learning without complex modifications.

Findings

Enhanced sample efficiency demonstrated across tasks

Minimal changes needed to existing algorithms

Backward trajectories provide stronger reward signals

Abstract

Current reinforcement learning algorithms train an agent using forward-generated trajectories, which provide little guidance so that the agent can explore as much as possible. While realizing the value of reinforcement learning results from sufficient exploration, this approach leads to a trade-off in losing sample efficiency, an essential factor impacting algorithm performance. Previous tasks use reward-shaping techniques and network structure modification to increase sample efficiency. However, these methods require many steps to implement. In this work, we propose novel backward curriculum reinforcement learning that begins training the agent using the backward trajectory of the episode instead of the original forward trajectory. This approach provides the agent with a strong reward signal, enabling more sample-efficient learning. Moreover, our method only requires a minor change in the algorithm of reversing the order of the trajectory before agent training, allowing a straightforward application to any state-of-the-art algorithm.