Dynamic Scene Novel View Synthesis via Deferred Spatio-temporal Consistency

TL;DR

This paper introduces a novel approach for dynamic scene view synthesis that combines deferred spatio-temporal consistency with a variational diffusion model, enabling robust rendering from casual captures without requiring perfect reconstructions.

Contribution

It proposes a deferred method that separates camera pose recovery from scene reconstruction and employs a variational diffusion process to enforce spatio-temporal consistency in dynamic scenes.

Findings

Effective in handling noisy, sparse reconstructions from casual captures.

Produces high-quality novel views without large dataset training.

Outperforms classic and recent learning-based baselines.

Abstract

Structure from motion (SfM) enables us to reconstruct a scene via casual capture from cameras at different viewpoints, and novel view synthesis (NVS) allows us to render a captured scene from a new viewpoint. Both are hard with casual capture and dynamic scenes: SfM produces noisy and spatio-temporally sparse reconstructed point clouds, resulting in NVS with spatio-temporally inconsistent effects. We consider SfM and NVS parts together to ease the challenge. First, for SfM, we recover stable camera poses, then we defer the requirement for temporally-consistent points across the scene and reconstruct only a sparse point cloud per timestep that is noisy in space-time. Second, for NVS, we present a variational diffusion formulation on depths and colors that lets us robustly cope with the noise by enforcing spatio-temporal consistency via per-pixel reprojection weights derived from the input views. Together, this deferred approach generates novel views for dynamic scenes without requiring challenging spatio-temporally consistent reconstructions nor training complex models on large datasets. We demonstrate our algorithm on real-world dynamic scenes against classic and more recent learning-based baseline approaches.