Viewpoint Textual Inversion: Discovering Scene Representations and 3D View Control in 2D Diffusion Models

TL;DR

This paper reveals that 2D diffusion models implicitly encode 3D scene representations, and introduces ViewNeTI, a method to discover and control 3D viewpoints in generated images, enabling advanced 3D vision applications.

Contribution

We propose ViewNeTI, a neural mapper that discovers 3D view tokens in diffusion models, enabling explicit control of 3D viewpoints in generated images.

Findings

The text latent space contains a continuous view-control manifold.

Evidence of a generalized view-control manifold across scenes.

State-of-the-art results in view-controlled generation and novel view synthesis.

Abstract

Text-to-image diffusion models generate impressive and realistic images, but do they learn to represent the 3D world from only 2D supervision? We demonstrate that yes, certain 3D scene representations are encoded in the text embedding space of models like Stable Diffusion. Our approach, Viewpoint Neural Textual Inversion (ViewNeTI), is to discover 3D view tokens; these tokens control the 3D viewpoint - the rendering pose in a scene - of generated images. Specifically, we train a small neural mapper to take continuous camera viewpoint parameters and predict a view token (a word embedding). This token conditions diffusion generation via cross-attention to produce images with the desired camera viewpoint. Using ViewNeTI as an evaluation tool, we report two findings: first, the text latent space has a continuous view-control manifold for particular 3D scenes; second, we find evidence for a generalized view-control manifold for all scenes. We conclude that since the view token controls the 3D `rendering' viewpoint, there is likely a scene representation embedded in frozen 2D diffusion models. Finally, we exploit the 3D scene representations for 3D vision tasks, namely, view-controlled text-to-image generation, and novel view synthesis from a single image, where our approach sets state-of-the-art for LPIPS. Code available at https://github.com/jmhb0/view_neti