VoyagerVision: Investigating the Role of Multi-modal Information for Open-ended Learning Systems

TL;DR

VoyagerVision explores the integration of visual inputs into open-ended learning systems, demonstrating that multi-modal data enhances the ability to interpret environments and generate complex structures in Minecraft, advancing towards more capable AGI models.

Contribution

This paper introduces VoyagerVision, a novel multi-modal model that uses visual feedback to improve open-ended task performance in environment construction, extending previous models like Voyager.

Findings

VoyagerVision created an average of 2.75 structures within fifty iterations.

It succeeded in half of the building unit tests in flat worlds.

Most failures occurred in complex structures.

Abstract

Open-endedness is an active field of research in the pursuit of capable Artificial General Intelligence (AGI), allowing models to pursue tasks of their own choosing. Simultaneously, recent advancements in Large Language Models (LLMs) such as GPT-4o [9] have allowed such models to be capable of interpreting image inputs. Implementations such as OMNI-EPIC [4] have made use of such features, providing an LLM with pixel data of an agent's POV to parse the environment and allow it to solve tasks. This paper proposes that providing these visual inputs to a model gives it greater ability to interpret spatial environments, and as such, can increase the number of tasks it can successfully perform, extending its open-ended potential. To this aim, this paper proposes VoyagerVision -- a multi-modal model capable of creating structures within Minecraft using screenshots as a form of visual feedback, building on the foundation of Voyager. VoyagerVision was capable of creating an average of 2.75 unique structures within fifty iterations of the system, as Voyager was incapable of this, it is an extension in an entirely new direction. Additionally, in a set of building unit tests VoyagerVision was successful in half of all attempts in flat worlds, with most failures arising in more complex structures. Project website is available at https://esmyth-dev.github.io/VoyagerVision.github.io/