Learning to Plan Optimistically: Uncertainty-Guided Deep Exploration via Latent Model Ensembles

TL;DR

This paper introduces LOVE, a novel exploration strategy for robot learning that combines latent models and ensemble-based uncertainty to improve long-term performance and sample efficiency in complex tasks.

Contribution

The paper proposes LOVE, a new method integrating latent models and ensemble uncertainty with UCB-based planning for deep exploration in robot control.

Findings

Over 20% sample efficiency improvement over state-of-the-art methods.

More than 30% improvement in sparse, hard-to-explore environments.

Effective in continuous action space visual control tasks.

Abstract

Learning complex robot behaviors through interaction requires structured exploration. Planning should target interactions with the potential to optimize long-term performance, while only reducing uncertainty where conducive to this objective. This paper presents Latent Optimistic Value Exploration (LOVE), a strategy that enables deep exploration through optimism in the face of uncertain long-term rewards. We combine latent world models with value function estimation to predict infinite-horizon returns and recover associated uncertainty via ensembling. The policy is then trained on an upper confidence bound (UCB) objective to identify and select the interactions most promising to improve long-term performance. We apply LOVE to visual robot control tasks in continuous action spaces and demonstrate on average more than 20% improved sample efficiency in comparison to state-of-the-art and other exploration objectives. In sparse and hard to explore environments we achieve an average improvement of over 30%.