ScenePainter: Semantically Consistent Perpetual 3D Scene Generation with Concept Relation Alignment

TL;DR

ScenePainter is a novel framework for perpetual 3D scene generation that maintains semantic consistency across long sequences by aligning scene concepts and dynamically refining relations, improving coherence and diversity.

Contribution

It introduces SceneConceptGraph for relation modeling and alignment, addressing semantic drift in long-range 3D scene synthesis, which is a significant advancement over existing methods.

Findings

Overcomes semantic drift in 3D scene generation

Produces more consistent and immersive 3D view sequences

Enhances diversity through dynamic relation refinement

Abstract

Perpetual 3D scene generation aims to produce long-range and coherent 3D view sequences, which is applicable for long-term video synthesis and 3D scene reconstruction. Existing methods follow a "navigate-and-imagine" fashion and rely on outpainting for successive view expansion. However, the generated view sequences suffer from semantic drift issue derived from the accumulated deviation of the outpainting module. To tackle this challenge, we propose ScenePainter, a new framework for semantically consistent 3D scene generation, which aligns the outpainter's scene-specific prior with the comprehension of the current scene. To be specific, we introduce a hierarchical graph structure dubbed SceneConceptGraph to construct relations among multi-level scene concepts, which directs the outpainter for consistent novel views and can be dynamically refined to enhance diversity. Extensive experiments demonstrate that our framework overcomes the semantic drift issue and generates more consistent and immersive 3D view sequences. Project Page: https://xiac20.github.io/ScenePainter/.