BetterScene: 3D Scene Synthesis with Representation-Aligned Generative Model

TL;DR

BetterScene enhances novel view synthesis from sparse photos by aligning diffusion model representations and integrating 3D Gaussian Splatting, resulting in more consistent, artifact-free 3D scene reconstructions.

Contribution

It introduces a novel approach combining diffusion model latent space regularizations with 3D Gaussian Splatting for improved 3D scene synthesis.

Findings

Outperforms state-of-the-art methods on DL3DV-10K dataset.

Produces continuous, artifact-free novel views.

Enhances view consistency in 3D scene synthesis.

Abstract

We present BetterScene, an approach to enhance novel view synthesis (NVS) quality for diverse real-world scenes using extremely sparse, unconstrained photos. BetterScene leverages the production-ready Stable Video Diffusion (SVD) model pretrained on billions of frames as a strong backbone, aiming to mitigate artifacts and recover view-consistent details at inference time. Conventional methods have developed similar diffusion-based solutions to address these challenges of novel view synthesis. Despite significant improvements, these methods typically rely on off-the-shelf pretrained diffusion priors and fine-tune only the UNet module while keeping other components frozen, which still leads to inconsistent details and artifacts even when incorporating geometry-aware regularizations like depth or semantic conditions. To address this, we investigate the latent space of the diffusion model and introduce two components: (1) temporal equivariance regularization and (2) vision foundation model-aligned representation, both applied to the variational autoencoder (VAE) module within the SVD pipeline. BetterScene integrates a feed-forward 3D Gaussian Splatting (3DGS) model to render features as inputs for the SVD enhancer and generate continuous, artifact-free, consistent novel views. We evaluate on the challenging DL3DV-10K dataset and demonstrate superior performance compared to state-of-the-art methods.