Generative World Explorer

TL;DR

Generative World Explorer (Genex) enables agents to mentally explore and revise beliefs about large-scale 3D environments using generated observations, improving decision-making without physical exploration.

Contribution

We introduce Genex, a novel framework for mental exploration in embodied AI, and create Genex-DB, a synthetic urban scene dataset for training and evaluation.

Findings

Genex generates high-quality, consistent observations during long-horizon exploration.

Belief updates with generated observations improve decision-making accuracy.

Genex enhances planning capabilities of existing AI agents.

Abstract

Planning with partial observation is a central challenge in embodied AI. A majority of prior works have tackled this challenge by developing agents that physically explore their environment to update their beliefs about the world state. In contrast, humans can $\textit{imagine}$ unseen parts of the world through a mental exploration and $\textit{revise}$ their beliefs with imagined observations. Such updated beliefs can allow them to make more informed decisions, without necessitating the physical exploration of the world at all times. To achieve this human-like ability, we introduce the $\textit{Generative World Explorer (Genex)}$, an egocentric world exploration framework that allows an agent to mentally explore a large-scale 3D world (e.g., urban scenes) and acquire imagined observations to update its belief. This updated belief will then help the agent to make a more informed decision at the current step. To train $\textit{Genex}$, we create a synthetic urban scene dataset, Genex-DB. Our experimental results demonstrate that (1) $\textit{Genex}$ can generate high-quality and consistent observations during long-horizon exploration of a large virtual physical world and (2) the beliefs updated with the generated observations can inform an existing decision-making model (e.g., an LLM agent) to make better plans.