STEVE Series: Step-by-Step Construction of Agent Systems in Minecraft

TL;DR

This paper introduces the STEVE series, a step-by-step approach to building embodied agent systems in Minecraft using large language models, achieving significant improvements in task performance and system complexity.

Contribution

The paper presents a novel hierarchical multi-agent system in Minecraft, combining LLMs with vision, memory, and knowledge distillation, advancing the construction methodology of embodied agents.

Findings

Agents outperform previous methods by 2.5 to 7.3 times in task efficiency.

The system successfully completes navigation and creative tasks in Minecraft.

Knowledge distillation effectively prunes the agent system for efficiency.

Abstract

Building an embodied agent system with a large language model (LLM) as its core is a promising direction. Due to the significant costs and uncontrollable factors associated with deploying and training such agents in the real world, we have decided to begin our exploration within the Minecraft environment. Our STEVE Series agents can complete basic tasks in a virtual environment and more challenging tasks such as navigation and even creative tasks, with an efficiency far exceeding previous state-of-the-art methods by a factor of $2.5\times$ to $7.3\times$. We begin our exploration with a vanilla large language model, augmenting it with a vision encoder and an action codebase trained on our collected high-quality dataset STEVE-21K. Subsequently, we enhanced it with a Critic and memory to transform it into a complex system. Finally, we constructed a hierarchical multi-agent system. Our recent work explored how to prune the agent system through knowledge distillation. In the future, we will explore more potential applications of STEVE agents in the real world.