Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenes

TL;DR

This paper introduces IAMMIR, a framework combining self-supervised visual representation learning with a novel intrinsic reward, MMIR, to improve sample efficiency in sparse reward visual navigation tasks.

Contribution

It proposes a new intrinsic reward, MMIR, and a combined visual representation method to enhance learning in sparse reward environments.

Findings

Achieves state-of-the-art sample efficiency in Vizdoom navigation.

At least 2 times faster convergence to 100% success rate.

Effective decomposition of visual representation and reward handling.

Abstract

Many scenes in real life can be abstracted to the sparse reward visual scenes, where it is difficult for an agent to tackle the task under the condition of only accepting images and sparse rewards. We propose to decompose this problem into two sub-problems: the visual representation and the sparse reward. To address them, a novel framework IAMMIR combining the self-supervised representation learning with the intrinsic motivation is presented. For visual representation, a representation driven by a combination of the imageaugmented forward dynamics and the reward is acquired. For sparse rewards, a new type of intrinsic reward is designed, the Momentum Memory Intrinsic Reward (MMIR). It utilizes the difference of the outputs from the current model (online network) and the historical model (target network) to present the agent's state familiarity. Our method is evaluated on the visual navigation task with sparse rewards in Vizdoom. Experiments demonstrate that our method achieves the state of the art performance in sample efficiency, at least 2 times faster than the existing methods reaching 100% success rate.