Generalized Back-Stepping Experience Replay in Sparse-Reward Environments

TL;DR

This paper introduces Generalized Back-Stepping Experience Replay (GBER), an extension of BER designed for sparse-reward environments, enhancing exploration and learning efficiency in complex, structured tasks.

Contribution

The paper proposes GBER, which extends BER with relabeling and diverse sampling strategies, enabling effective learning in sparse, structured environments.

Findings

GBER significantly improves performance in maze navigation tasks.

GBER enhances stability and learning efficiency in sparse-reward settings.

Experimental results show GBER outperforms baseline algorithms.

Abstract

Back-stepping experience replay (BER) is a reinforcement learning technique that can accelerate learning efficiency in reversible environments. BER trains an agent with generated back-stepping transitions of collected experiences and normal forward transitions. However, the original algorithm is designed for a dense-reward environment that does not require complex exploration, limiting the BER technique to demonstrate its full potential. Herein, we propose an enhanced version of BER called Generalized BER (GBER), which extends the original algorithm to sparse-reward environments, particularly those with complex structures that require the agent to explore. GBER improves the performance of BER by introducing relabeling mechanism and applying diverse sampling strategies. We evaluate our modified version, which is based on a goal-conditioned deep deterministic policy gradient offline learning algorithm, across various maze navigation environments. The experimental results indicate that the GBER algorithm can significantly boost the performance and stability of the baseline algorithm in various sparse-reward environments, especially those with highly structural symmetricity.