Video Perception Models for 3D Scene Synthesis

TL;DR

VIPScene is a novel framework that combines video generation, 3D reconstruction, and perception models to synthesize realistic, coherent 3D scenes from text or image prompts, leveraging commonsense knowledge of the physical world.

Contribution

The paper introduces VIPScene, a new approach that integrates video perception models with 3D scene synthesis, improving realism and structural consistency over existing methods.

Findings

VIPScene outperforms existing 3D scene synthesis methods.

The framework achieves high realism and structural coherence.

First-Person View Score effectively evaluates scene plausibility.

Abstract

Traditionally, 3D scene synthesis requires expert knowledge and significant manual effort. Automating this process could greatly benefit fields such as architectural design, robotics simulation, virtual reality, and gaming. Recent approaches to 3D scene synthesis often rely on the commonsense reasoning of large language models (LLMs) or strong visual priors of modern image generation models. However, current LLMs demonstrate limited 3D spatial reasoning ability, which restricts their ability to generate realistic and coherent 3D scenes. Meanwhile, image generation-based methods often suffer from constraints in viewpoint selection and multi-view inconsistencies. In this work, we present Video Perception models for 3D Scene synthesis (VIPScene), a novel framework that exploits the encoded commonsense knowledge of the 3D physical world in video generation models to ensure coherent scene layouts and consistent object placements across views. VIPScene accepts both text and image prompts and seamlessly integrates video generation, feedforward 3D reconstruction, and open-vocabulary perception models to semantically and geometrically analyze each object in a scene. This enables flexible scene synthesis with high realism and structural consistency. For more precise analysis, we further introduce First-Person View Score (FPVScore) for coherence and plausibility evaluation, utilizing continuous first-person perspective to capitalize on the reasoning ability of multimodal large language models. Extensive experiments show that VIPScene significantly outperforms existing methods and generalizes well across diverse scenarios. The code will be released.