Semi-supervised Synthesis of High-Resolution Editable Textures for 3D Humans

TL;DR

This paper presents a semi-supervised method for generating high-resolution, editable textures for 3D human models, allowing style control and layout independence, useful for AR/VR applications.

Contribution

It introduces ReAVAE, a novel region-adaptive adversarial variational autoencoder, and a data augmentation technique for improved texture synthesis.

Findings

Outperforms prior methods in texture quality

Enables independent style and layout control

Supports style mixing for diverse textures

Abstract

We introduce a novel approach to generate diverse high fidelity texture maps for 3D human meshes in a semi-supervised setup. Given a segmentation mask defining the layout of the semantic regions in the texture map, our network generates high-resolution textures with a variety of styles, that are then used for rendering purposes. To accomplish this task, we propose a Region-adaptive Adversarial Variational AutoEncoder (ReAVAE) that learns the probability distribution of the style of each region individually so that the style of the generated texture can be controlled by sampling from the region-specific distributions. In addition, we introduce a data generation technique to augment our training set with data lifted from single-view RGB inputs. Our training strategy allows the mixing of reference image styles with arbitrary styles for different regions, a property which can be valuable for virtual try-on AR/VR applications. Experimental results show that our method synthesizes better texture maps compared to prior work while enabling independent layout and style controllability.