SemUV: Deep Learning based semantic manipulation over UV texture map of virtual human heads

TL;DR

SemUV introduces a deep learning method for semantic editing of UV texture maps of virtual human heads, enabling precise, identity-preserving modifications in 3D graphics pipelines using StyleGAN trained on UV data.

Contribution

It presents a novel approach for semantic manipulation directly in UV texture space, leveraging StyleGAN and a boundary-based method for intuitive editing.

Findings

Outperforms 2D manipulation in preserving identity.

Effectively modifies age, gender, facial hair features.

Simple and compatible with standard 3D pipelines.

Abstract

Designing and manipulating virtual human heads is essential across various applications, including AR, VR, gaming, human-computer interaction and VFX. Traditional graphic-based approaches require manual effort and resources to achieve accurate representation of human heads. While modern deep learning techniques can generate and edit highly photorealistic images of faces, their focus remains predominantly on 2D facial images. This limitation makes them less suitable for 3D applications. Recognizing the vital role of editing within the UV texture space as a key component in the 3D graphics pipeline, our work focuses on this aspect to benefit graphic designers by providing enhanced control and precision in appearance manipulation. Research on existing methods within the UV texture space is limited, complex, and poses challenges. In this paper, we introduce SemUV: a simple and effective approach using the FFHQ-UV dataset for semantic manipulation directly within the UV texture space. We train a StyleGAN model on the publicly available FFHQ-UV dataset, and subsequently train a boundary for interpolation and semantic feature manipulation. Through experiments comparing our method with 2D manipulation technique, we demonstrate its superior ability to preserve identity while effectively modifying semantic features such as age, gender, and facial hair. Our approach is simple, agnostic to other 3D components such as structure, lighting, and rendering, and also enables seamless integration into standard 3D graphics pipelines without demanding extensive domain expertise, time, or resources.