Optimistic World Models: Efficient Exploration in Model-Based Deep Reinforcement Learning

TL;DR

This paper introduces Optimistic World Models (OWMs), a scalable deep RL exploration method that biases model learning towards higher rewards, improving sample efficiency and performance in sparse-reward environments.

Contribution

The paper presents OWMs, a novel, fully gradient-based optimistic exploration framework integrated into deep world models, enhancing exploration without uncertainty estimation.

Findings

OWMs improve sample efficiency in sparse-reward tasks.

Optimistic DreamerV3 and STORM outperform baseline models.

The approach requires minimal modifications to existing frameworks.

Abstract

Efficient exploration remains a central challenge in reinforcement learning (RL), particularly in sparse-reward environments. We introduce Optimistic World Models (OWMs), a principled and scalable framework for optimistic exploration that brings classical reward-biased maximum likelihood estimation (RBMLE) from adaptive control into deep RL. In contrast to upper confidence bound (UCB)-style exploration methods, OWMs incorporate optimism directly into model learning by augmentation with an optimistic dynamics loss that biases imagined transitions toward higher-reward outcomes. This fully gradient-based loss requires neither uncertainty estimates nor constrained optimization. Our approach is plug-and-play with existing world model frameworks, preserving scalability while requiring only minimal modifications to standard training procedures. We instantiate OWMs within two state-of-the-art world model architectures, leading to Optimistic DreamerV3 and Optimistic STORM, which demonstrate significant improvements in sample efficiency and cumulative return compared to their baseline counterparts.