3D-Belief: Embodied Belief Inference via Generative 3D World Modeling

TL;DR

This paper introduces 3D-Belief, a generative 3D world model that maintains and updates an agent's beliefs about unobserved environments, enhancing embodied reasoning and navigation under partial observability.

Contribution

The work presents a novel 3D belief inference framework that explicitly models uncertainty and updates beliefs online, improving scene understanding and task performance in embodied agents.

Findings

3D-Belief improves scene memory and imagination quality.

It enhances object navigation in simulation and real-world environments.

Outperforms state-of-the-art methods in 3D belief inference and embodied tasks.

Abstract

Recent advances in visual generative models have highlighted the promise of learning generative world models. However, most existing approaches frame world modeling as novel-view synthesis or future-frame prediction, emphasizing visual realism rather than the structured uncertainty required by embodied agents acting under partial observability. In this work, we propose a different perspective: world modeling as embodied belief inference in 3D space. From this view, a world model should not merely render what may be seen, but maintain and update an agent's belief about the unobserved 3D world as new observations are acquired. We identify several key capabilities for such models, including spatially consistent scene memory, multi-hypothesis belief sampling, sequential belief updating, and semantically informed prediction of unseen regions. We instantiate these ideas in 3D-Belief, a generative 3D world model that infers explicit, actionable 3D beliefs from partial observations and updates them online over time. Unlike prior visual prediction models, 3D-Belief represents uncertainty directly in 3D, enabling embodied agents to imagine plausible scene completions and reason over partially observed environments. We evaluate 3D-Belief on 2D visual quality for scene memory and unobserved-scene imagination, object- and scene-level 3D imagination using our proposed 3D-CORE benchmark, and challenging object navigation tasks in both simulation and the real world. Experiments show that 3D-Belief improves 2D and 3D imagination quality and downstream embodied task performance compared to state-of-the-art methods.