Curious Replay for Model-based Adaptation

TL;DR

This paper introduces Curious Replay, a curiosity-driven prioritized experience replay method that enhances model-based reinforcement learning agents' ability to adapt quickly and improves their performance on exploration tasks and benchmarks.

Contribution

The paper proposes Curious Replay, a novel prioritized experience replay technique tailored for model-based agents, significantly improving adaptation and exploration performance.

Findings

DreamerV3 with Curious Replay achieves a mean score of 19.4 on Crafter.

Curious Replay outperforms uniform replay in Crafter benchmark.

Maintains similar performance on Deepmind Control Suite.

Abstract

Agents must be able to adapt quickly as an environment changes. We find that existing model-based reinforcement learning agents are unable to do this well, in part because of how they use past experiences to train their world model. Here, we present Curious Replay -- a form of prioritized experience replay tailored to model-based agents through use of a curiosity-based priority signal. Agents using Curious Replay exhibit improved performance in an exploration paradigm inspired by animal behavior and on the Crafter benchmark. DreamerV3 with Curious Replay surpasses state-of-the-art performance on Crafter, achieving a mean score of 19.4 that substantially improves on the previous high score of 14.5 by DreamerV3 with uniform replay, while also maintaining similar performance on the Deepmind Control Suite. Code for Curious Replay is available at https://github.com/AutonomousAgentsLab/curiousreplay