CtrlNeRF: The Generative Neural Radiation Fields for the Controllable Synthesis of High-fidelity 3D-Aware Images

TL;DR

CtrlNeRF introduces a controllable generative model using a shared MLP to produce high-fidelity, 3D-consistent images with adjustable shape and appearance, enabling novel view synthesis beyond training data.

Contribution

It presents a novel single-MLP framework for multi-scene 3D-aware image generation with controllable shape and appearance manipulation.

Findings

Outperforms existing methods in 3D-aware image generation

Enables precise control over shape and appearance during synthesis

Allows synthesis of novel views through camera pose and feature interpolation

Abstract

The neural radiance field (NERF) advocates learning the continuous representation of 3D geometry through a multilayer perceptron (MLP). By integrating this into a generative model, the generative neural radiance field (GRAF) is capable of producing images from random noise z without 3D supervision. In practice, the shape and appearance are modeled by z_s and z_a, respectively, to manipulate them separately during inference. However, it is challenging to represent multiple scenes using a solitary MLP and precisely control the generation of 3D geometry in terms of shape and appearance. In this paper, we introduce a controllable generative model (i.e. \textbf{CtrlNeRF}) that uses a single MLP network to represent multiple scenes with shared weights. Consequently, we manipulated the shape and appearance codes to realize the controllable generation of high-fidelity images with 3D consistency. Moreover, the model enables the synthesis of novel views that do not exist in the training sets via camera pose alteration and feature interpolation. Extensive experiments were conducted to demonstrate its superiority in 3D-aware image generation compared to its counterparts.