Novel View Synthesis using DDIM Inversion

TL;DR

This paper introduces a lightweight, view translation framework using DDIM inversion and a specialized U-Net to synthesize high-fidelity novel views from a single image, overcoming limitations of existing methods.

Contribution

It proposes a novel fusion strategy and a camera pose-conditioned translation U-Net that improve detail preservation and generalization in single-image view synthesis.

Findings

Outperforms existing methods on MVImgNet dataset

Preserves texture and fine details effectively

Reduces blurriness in reconstructed views

Abstract

Synthesizing novel views from a single input image is a challenging task. It requires extrapolating the 3D structure of a scene while inferring details in occluded regions, and maintaining geometric consistency across viewpoints. Many existing methods must fine-tune large diffusion backbones using multiple views or train a diffusion model from scratch, which is extremely expensive. Additionally, they suffer from blurry reconstruction and poor generalization. This gap presents the opportunity to explore an explicit lightweight view translation framework that can directly utilize the high-fidelity generative capabilities of a pretrained diffusion model while reconstructing a scene from a novel view. Given the DDIM-inverted latent of a single input image, we employ a camera pose-conditioned translation U-Net, TUNet, to predict the inverted latent corresponding to the desired target view. However, the image sampled using the predicted latent may result in a blurry reconstruction. To this end, we propose a novel fusion strategy that exploits the inherent noise correlation structure observed in DDIM inversion. The proposed fusion strategy helps preserve the texture and fine-grained details. To synthesize the novel view, we use the fused latent as the initial condition for DDIM sampling, leveraging the generative prior of the pretrained diffusion model. Extensive experiments on MVImgNet demonstrate that our method outperforms existing methods.