DT-NVS: Diffusion Transformers for Novel View Synthesis

TL;DR

This paper introduces DT-NVS, a 3D diffusion model with transformer architecture for generalized novel view synthesis from a single image, trained on real-world videos, outperforming existing methods in diversity and quality.

Contribution

The paper presents a novel 3D diffusion model with transformer backbone, new camera conditioning strategies, and a unique training paradigm for real-world, unaligned datasets.

Findings

Outperforms state-of-the-art 3D diffusion models.

Generates diverse and high-quality novel views.

Effective on real-world, unaligned video datasets.

Abstract

Generating novel views of a natural scene, e.g., every-day scenes both indoors and outdoors, from a single view is an under-explored problem, even though it is an organic extension to the object-centric novel view synthesis. Existing diffusion-based approaches focus rather on small camera movements in real scenes or only consider unnatural object-centric scenes, limiting their potential applications in real-world settings. In this paper we move away from these constrained regimes and propose a 3D diffusion model trained with image-only losses on a large-scale dataset of real-world, multi-category, unaligned, and casually acquired videos of everyday scenes. We propose DT-NVS, a 3D-aware diffusion model for generalized novel view synthesis that exploits a transformer-based architecture backbone. We make significant contributions to transformer and self-attention architectures to translate images to 3d representations, and novel camera conditioning strategies to allow training on real-world unaligned datasets. In addition, we introduce a novel training paradigm swapping the role of reference frame between the conditioning image and the sampled noisy input. We evaluate our approach on the 3D task of generalized novel view synthesis from a single input image and show improvements over state-of-the-art 3D aware diffusion models and deterministic approaches, while generating diverse outputs.