CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

TL;DR

This paper introduces CAMPARI, a 3D-aware generative model that jointly learns camera and scene representations, enabling realistic image synthesis with explicit camera control and scene disentanglement from raw image collections.

Contribution

It proposes a novel approach to 3D-aware image synthesis by jointly learning camera generation and scene decomposition, improving realism and control.

Findings

Successfully learns camera and scene distributions from raw images.

Produces high-quality, 3D-consistent images with explicit camera control.

Achieves disentangled background and foreground representations.

Abstract

Tremendous progress in deep generative models has led to photorealistic image synthesis. While achieving compelling results, most approaches operate in the two-dimensional image domain, ignoring the three-dimensional nature of our world. Several recent works therefore propose generative models which are 3D-aware, i.e., scenes are modeled in 3D and then rendered differentiably to the image plane. This leads to impressive 3D consistency, but incorporating such a bias comes at a price: the camera needs to be modeled as well. Current approaches assume fixed intrinsics and a predefined prior over camera pose ranges. As a result, parameter tuning is typically required for real-world data, and results degrade if the data distribution is not matched. Our key hypothesis is that learning a camera generator jointly with the image generator leads to a more principled approach to 3D-aware image synthesis. Further, we propose to decompose the scene into a background and foreground model, leading to more efficient and disentangled scene representations. While training from raw, unposed image collections, we learn a 3D- and camera-aware generative model which faithfully recovers not only the image but also the camera data distribution. At test time, our model generates images with explicit control over the camera as well as the shape and appearance of the scene.