Semantic 3D-aware Portrait Synthesis and Manipulation Based on Compositional Neural Radiance Field

TL;DR

This paper introduces CNeRF, a novel neural radiance field approach that enables fine-grained semantic editing and manipulation of 3D-aware portrait synthesis by dividing images into semantic regions and learning independent radiance fields.

Contribution

The paper proposes a compositional NeRF that allows independent semantic region editing and decouples shape and texture within regions, enhancing 3D-aware portrait manipulation.

Findings

Enables independent manipulation of semantic regions.

Maintains high-quality 3D-consistent synthesis.

Demonstrates effectiveness through ablation studies and applications.

Abstract

Recently 3D-aware GAN methods with neural radiance field have developed rapidly. However, current methods model the whole image as an overall neural radiance field, which limits the partial semantic editability of synthetic results. Since NeRF renders an image pixel by pixel, it is possible to split NeRF in the spatial dimension. We propose a Compositional Neural Radiance Field (CNeRF) for semantic 3D-aware portrait synthesis and manipulation. CNeRF divides the image by semantic regions and learns an independent neural radiance field for each region, and finally fuses them and renders the complete image. Thus we can manipulate the synthesized semantic regions independently, while fixing the other parts unchanged. Furthermore, CNeRF is also designed to decouple shape and texture within each semantic region. Compared to state-of-the-art 3D-aware GAN methods, our approach enables fine-grained semantic region manipulation, while maintaining high-quality 3D-consistent synthesis. The ablation studies show the effectiveness of the structure and loss function used by our method. In addition real image inversion and cartoon portrait 3D editing experiments demonstrate the application potential of our method.