SAT3D: Image-driven Semantic Attribute Transfer in 3D

TL;DR

SAT3D introduces a novel 3D-aware semantic attribute transfer method that leverages style space editing, descriptor-based guidance, and CLIP-based quantitative measurement to achieve precise, photo-realistic attribute manipulation from reference images.

Contribution

The paper presents a new 3D-aware attribute transfer approach using style space editing and a CLIP-based measurement module, enabling more accurate and customizable semantic attribute transfer in 3D images.

Findings

Effective transfer of semantic attributes across multiple domains.

Outperforms classical 2D image editing methods in accuracy and realism.

Demonstrates the method's flexibility and potential for diverse applications.

Abstract

GAN-based image editing task aims at manipulating image attributes in the latent space of generative models. Most of the previous 2D and 3D-aware approaches mainly focus on editing attributes in images with ambiguous semantics or regions from a reference image, which fail to achieve photographic semantic attribute transfer, such as the beard from a photo of a man. In this paper, we propose an image-driven Semantic Attribute Transfer method in 3D (SAT3D) by editing semantic attributes from a reference image. For the proposed method, the exploration is conducted in the style space of a pre-trained 3D-aware StyleGAN-based generator by learning the correlations between semantic attributes and style code channels. For guidance, we associate each attribute with a set of phrase-based descriptor groups, and develop a Quantitative Measurement Module (QMM) to quantitatively describe the attribute characteristics in images based on descriptor groups, which leverages the image-text comprehension capability of CLIP. During the training process, the QMM is incorporated into attribute losses to calculate attribute similarity between images, guiding target semantic transferring and irrelevant semantics preserving. We present our 3D-aware attribute transfer results across multiple domains and also conduct comparisons with classical 2D image editing methods, demonstrating the effectiveness and customizability of our SAT3D.