Beyond Surprise: Improving Exploration Through Surprise Novelty
Hung Le, Kien Do, Dung Nguyen, Svetha Venkatesh

TL;DR
This paper introduces a novel reinforcement learning model that enhances exploration by measuring the novelty of surprise through a memory network, leading to improved performance in sparse reward environments.
Contribution
It proposes a surprise memory mechanism that captures surprise novelty, addressing limitations of existing surprise-driven exploration methods.
Findings
Enhanced exploration efficiency in sparse reward settings
Significant performance improvements in Atari games
Effective reduction of attraction to noisy observations
Abstract
We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors of a memory network wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based intrinsic motivators, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experiments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final performance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Artificial Intelligence in Games · Domain Adaptation and Few-Shot Learning
MethodsMemory Network
