Novel View Synthesis from A Few Glimpses via Test-Time Natural Video Completion

TL;DR

This paper introduces a zero-shot method for novel view synthesis from sparse inputs by using pretrained video diffusion models to hallucinate intermediate views, enhancing 3D scene reconstruction without scene-specific training.

Contribution

It presents a test-time natural video completion approach that leverages pretrained diffusion models and an iterative feedback loop to improve scene reconstruction and view synthesis from minimal input views.

Findings

Outperforms strong 3D-GS baselines on multiple datasets under extreme sparsity.

Produces coherent, high-fidelity renderings without scene-specific training.

Effectively densifies supervision for 3D scene reconstruction.

Abstract

Given just a few glimpses of a scene, can you imagine the movie playing out as the camera glides through it? That's the lens we take on \emph{sparse-input novel view synthesis}, not only as filling spatial gaps between widely spaced views, but also as \emph{completing a natural video} unfolding through space. We recast the task as \emph{test-time natural video completion}, using powerful priors from \emph{pretrained video diffusion models} to hallucinate plausible in-between views. Our \emph{zero-shot, generation-guided} framework produces pseudo views at novel camera poses, modulated by an \emph{uncertainty-aware mechanism} for spatial coherence. These synthesized frames densify supervision for \emph{3D Gaussian Splatting} (3D-GS) for scene reconstruction, especially in under-observed regions. An iterative feedback loop lets 3D geometry and 2D view synthesis inform each other, improving both the scene reconstruction and the generated views. The result is coherent, high-fidelity renderings from sparse inputs \emph{without any scene-specific training or fine-tuning}. On LLFF, DTU, DL3DV, and MipNeRF-360, our method significantly outperforms strong 3D-GS baselines under extreme sparsity.