FVGen: Accelerating Novel-View Synthesis with Adversarial Video Diffusion Distillation

TL;DR

FVGen introduces a fast, efficient framework for novel view synthesis from sparse images by distilling video diffusion models into a few-step model, greatly reducing sampling time while maintaining high visual quality.

Contribution

The paper proposes a novel video diffusion model distillation method using GANs to enable rapid novel view synthesis with significantly fewer sampling steps.

Findings

Reduces sampling time by over 90% compared to previous methods.

Maintains or improves visual quality of generated views.

Enhances efficiency in 3D reconstruction from sparse views.

Abstract

Recent progress in 3D reconstruction has enabled realistic 3D models from dense image captures, yet challenges persist with sparse views, often leading to artifacts in unseen areas. Recent works leverage Video Diffusion Models (VDMs) to generate dense observations, filling the gaps when only sparse views are available for 3D reconstruction tasks. A significant limitation of these methods is their slow sampling speed when using VDMs. In this paper, we present FVGen, a novel framework that addresses this challenge by enabling fast novel view synthesis using VDMs in as few as four sampling steps. We propose a novel video diffusion model distillation method that distills a multi-step denoising teacher model into a few-step denoising student model using Generative Adversarial Networks (GANs) and softened reverse KL-divergence minimization. Extensive experiments on real-world datasets show that, compared to previous works, our framework generates the same number of novel views with similar (or even better) visual quality while reducing sampling time by more than 90%. FVGen significantly improves time efficiency for downstream reconstruction tasks, particularly when working with sparse input views (more than 2) where pre-trained VDMs need to be run multiple times to achieve better spatial coverage.