Introspective Experience Replay: Look Back When Surprised
Ramnath Kumar, Dheeraj Nagaraj

TL;DR
This paper introduces Introspective Experience Replay (IER), a novel sampling method in reinforcement learning that improves convergence and reliability by selectively revisiting surprising experiences, outperforming existing replay techniques.
Contribution
The paper proposes IER, a new experience replay strategy that builds on reverse experience replay, tailored for neural networks, enhancing RL performance and stability.
Findings
IER outperforms UER, PER, and HER in most tasks.
IER provides more reliable convergence with neural function approximation.
Empirical results show superior performance of IER across various RL benchmarks.
Abstract
In reinforcement learning (RL), experience replay-based sampling techniques play a crucial role in promoting convergence by eliminating spurious correlations. However, widely used methods such as uniform experience replay (UER) and prioritized experience replay (PER) have been shown to have sub-optimal convergence and high seed sensitivity respectively. To address these issues, we propose a novel approach called IntrospectiveExperience Replay (IER) that selectively samples batches of data points prior to surprising events. Our method builds upon the theoretically sound reverse experience replay (RER) technique, which has been shown to reduce bias in the output of Q-learning-type algorithms with linear function approximation. However, this approach is not always practical or reliable when using neural function approximation. Through empirical evaluations, we demonstrate that IER with…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMental Health Research Topics · Behavioral Health and Interventions · Functional Brain Connectivity Studies
MethodsPrioritized Experience Replay · Experience Replay
