Benchmarking and Analyzing 3D-aware Image Synthesis with a Modularized Codebase

TL;DR

This paper introduces Carver, a modular codebase for 3D-aware image synthesis that enables fair comparison of different approaches and provides in-depth analysis of key components to advance understanding in the field.

Contribution

The paper presents a modularized, well-structured codebase for 3D-aware image synthesis, allowing independent development and comparison of various modules.

Findings

Reproduction of state-of-the-art algorithms using Carver

Insights into the impact of point feature types and generator components

Analysis of the reliance on camera pose prior

Abstract

Despite the rapid advance of 3D-aware image synthesis, existing studies usually adopt a mixture of techniques and tricks, leaving it unclear how each part contributes to the final performance in terms of generality. Following the most popular and effective paradigm in this field, which incorporates a neural radiance field (NeRF) into the generator of a generative adversarial network (GAN), we build a well-structured codebase, dubbed Carver, through modularizing the generation process. Such a design allows researchers to develop and replace each module independently, and hence offers an opportunity to fairly compare various approaches and recognize their contributions from the module perspective. The reproduction of a range of cutting-edge algorithms demonstrates the availability of our modularized codebase. We also perform a variety of in-depth analyses, such as the comparison across different types of point feature, the necessity of the tailing upsampler in the generator, the reliance on the camera pose prior, etc., which deepen our understanding of existing methods and point out some further directions of the research work. We release code and models at https://github.com/qiuyu96/Carver to facilitate the development and evaluation of this field.