Dynamic View Synthesis as an Inverse Problem

TL;DR

This paper presents a novel, training-free approach for dynamic view synthesis from monocular videos by manipulating the noise initialization in a pre-trained diffusion model, enabling high-fidelity results without additional training.

Contribution

It introduces a new noise representation and a visibility-aware sampling method, allowing deterministic inversion and dynamic view synthesis without weight updates or auxiliary modules.

Findings

Effective dynamic view synthesis achieved through structured latent manipulation.

Introduces K-order Recursive Noise Representation for precise latent alignment.

Demonstrates high-fidelity results in monocular video scenarios.

Abstract

In this work, we address dynamic view synthesis from monocular videos as an inverse problem in a training-free setting. By redesigning the noise initialization phase of a pre-trained video diffusion model, we enable high-fidelity dynamic view synthesis without any weight updates or auxiliary modules. We begin by identifying a fundamental obstacle to deterministic inversion arising from zero-terminal signal-to-noise ratio (SNR) schedules and resolve it by introducing a novel noise representation, termed K-order Recursive Noise Representation. We derive a closed form expression for this representation, enabling precise and efficient alignment between the VAE-encoded and the DDIM inverted latents. To synthesize newly visible regions resulting from camera motion, we introduce Stochastic Latent Modulation, which performs visibility aware sampling over the latent space to complete occluded regions. Comprehensive experiments demonstrate that dynamic view synthesis can be effectively performed through structured latent manipulation in the noise initialization phase.