ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting

TL;DR

ROCKET-1 introduces a visual-temporal context prompting method that enhances vision-language models with spatial reasoning capabilities, enabling complex open-world interactions in Minecraft with significant performance improvements.

Contribution

The paper presents a novel communication protocol and training approach that allows VLMs to incorporate spatial and temporal context for improved decision-making in embodied environments.

Findings

Achieved 76% improvement in open-world interaction performance in Minecraft.

Enabled VLM-based agents to perform complex spatial reasoning tasks.

Demonstrated the effectiveness of visual-temporal context prompting in real-time environments.

Abstract

Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to embodied decision-making in open-world environments presents challenges. One critical issue is bridging the gap between discrete entities in low-level observations and the abstract concepts required for effective planning. A common solution is building hierarchical agents, where VLMs serve as high-level reasoners that break down tasks into executable sub-tasks, typically specified using language. However, language suffers from the inability to communicate detailed spatial information. We propose visual-temporal context prompting, a novel communication protocol between VLMs and policy models. This protocol leverages object segmentation from past observations to guide policy-environment interactions. Using this approach, we train ROCKET-1, a low-level policy that predicts actions based on concatenated visual observations and segmentation masks, supported by real-time object tracking from SAM-2. Our method unlocks the potential of VLMs, enabling them to tackle complex tasks that demand spatial reasoning. Experiments in Minecraft show that our approach enables agents to achieve previously unattainable tasks, with a $\mathbf{76}\%$ absolute improvement in open-world interaction performance. Codes and demos are now available on the project page: https://craftjarvis.github.io/ROCKET-1.