RoamScene3D: Immersive Text-to-3D Scene Generation via Adaptive Object-aware Roaming

TL;DR

RoamScene3D introduces a novel framework for generating immersive, photorealistic 3D scenes from text by leveraging semantic scene graphs and adaptive camera trajectories, overcoming spatial blindness and inpainting limitations.

Contribution

The paper presents a new approach combining semantic scene graph reasoning with motion-aware inpainting to improve 3D scene generation from text, addressing key limitations of prior methods.

Findings

Outperforms state-of-the-art in scene consistency and realism

Effectively models object relations for adaptive scene exploration

Demonstrates robustness to camera motion and occlusions

Abstract

Generating immersive 3D scenes from texts is a core task in computer vision, crucial for applications in virtual reality and game development. Despite the promise of leveraging 2D diffusion priors, existing methods suffer from spatial blindness and rely on predefined trajectories that fail to exploit the inner relationships among salient objects. Consequently, these approaches are unable to comprehend the semantic layout, preventing them from exploring the scene adaptively to infer occluded content. Moreover, current inpainting models operate in 2D image space, struggling to plausibly fill holes caused by camera motion. To address these limitations, we propose RoamScene3D, a novel framework that bridges the gap between semantic guidance and spatial generation. Our method reasons about the semantic relations among objects and produces consistent and photorealistic scenes. Specifically, we employ a vision-language model (VLM) to construct a scene graph that encodes object relations, guiding the camera to perceive salient object boundaries and plan an adaptive roaming trajectory. Furthermore, to mitigate the limitations of static 2D priors, we introduce a Motion-Injected Inpainting model that is fine-tuned on a synthetic panoramic dataset integrating authentic camera trajectories, making it adaptive to camera motion. Extensive experiments demonstrate that with semantic reasoning and geometric constraints, our method significantly outperforms state-of-the-art approaches in producing consistent and photorealistic scenes. Our code is available at https://github.com/JS-CHU/RoamScene3D.