SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations

TL;DR

SpatialCrafter leverages video diffusion models and geometric constraints to enable photorealistic 3D scene reconstruction from limited or single-view inputs, advancing the field of novel view synthesis.

Contribution

It introduces a novel framework combining diffusion models, geometric constraints, and depth priors for effective sparse-view 3D scene reconstruction.

Findings

Improves 3D scene reconstruction from sparse views.

Generates realistic 3D scene appearances.

Achieves precise camera control and 3D consistency.

Abstract

Novel view synthesis (NVS) boosts immersive experiences in computer vision and graphics. Existing techniques, though progressed, rely on dense multi-view observations, restricting their application. This work takes on the challenge of reconstructing photorealistic 3D scenes from sparse or single-view inputs. We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations, thereby alleviating reconstruction ambiguity. Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency, further reinforced by a unified scale estimation strategy to handle scale discrepancies across datasets. Furthermore, by integrating monocular depth priors with semantic features in the video latent space, our framework directly regresses 3D Gaussian primitives and efficiently processes long-sequence features using a hybrid network structure. Extensive experiments show our method enhances sparse view reconstruction and restores the realistic appearance of 3D scenes.