AdaMemento: Adaptive Memory-Assisted Policy Optimization for Reinforcement Learning

TL;DR

AdaMemento introduces an adaptive memory framework for reinforcement learning that leverages both positive and negative experiences, guided by intrinsic motivation and ensemble learning, to improve exploration and policy optimization in sparse reward environments.

Contribution

It proposes a novel memory-reflection module and intrinsic motivation paradigm, enhancing experience filtering and exploration in RL beyond simple memory storage.

Findings

Achieves significant performance improvements over previous methods.

Effectively distinguishes subtle states for better exploration.

Theoretically proven advantages of intrinsic motivation and ensemble mechanisms.

Abstract

In sparse reward scenarios of reinforcement learning (RL), the memory mechanism provides promising shortcuts to policy optimization by reflecting on past experiences like humans. However, current memory-based RL methods simply store and reuse high-value policies, lacking a deeper refining and filtering of diverse past experiences and hence limiting the capability of memory. In this paper, we propose AdaMemento, an adaptive memory-enhanced RL framework. Instead of just memorizing positive past experiences, we design a memory-reflection module that exploits both positive and negative experiences by learning to predict known local optimal policies based on real-time states. To effectively gather informative trajectories for the memory, we further introduce a fine-grained intrinsic motivation paradigm, where nuances in similar states can be precisely distinguished to guide exploration. The exploitation of past experiences and exploration of new policies are then adaptively coordinated by ensemble learning to approach the global optimum. Furthermore, we theoretically prove the superiority of our new intrinsic motivation and ensemble mechanism. From 59 quantitative and visualization experiments, we confirm that AdaMemento can distinguish subtle states for better exploration and effectively exploiting past experiences in memory, achieving significant improvement over previous methods.