Open-World Reinforcement Learning over Long Short-Term Imagination

TL;DR

This paper introduces LS-Imagine, a method that extends the imagination horizon of world models in visual reinforcement learning, enabling better exploration and long-term decision-making in open worlds.

Contribution

We propose a long short-term world model that simulates goal-conditioned jumpy transitions, improving exploration in high-dimensional open-world environments.

Findings

Outperforms state-of-the-art methods in MineDojo

Enhances exploration efficiency for long-horizon tasks

Integrates long-term values into behavior learning

Abstract

Training visual reinforcement learning agents in a high-dimensional open world presents significant challenges. While various model-based methods have improved sample efficiency by learning interactive world models, these agents tend to be "short-sighted", as they are typically trained on short snippets of imagined experiences. We argue that the primary challenge in open-world decision-making is improving the exploration efficiency across a vast state space, especially for tasks that demand consideration of long-horizon payoffs. In this paper, we present LS-Imagine, which extends the imagination horizon within a limited number of state transition steps, enabling the agent to explore behaviors that potentially lead to promising long-term feedback. The foundation of our approach is to build a $\textit{long short-term world model}$. To achieve this, we simulate goal-conditioned jumpy state transitions and compute corresponding affordance maps by zooming in on specific areas within single images. This facilitates the integration of direct long-term values into behavior learning. Our method demonstrates significant improvements over state-of-the-art techniques in MineDojo.