AUV-Net: Learning Aligned UV Maps for Texture Transfer and Synthesis

TL;DR

AUV-Net learns to embed 3D surfaces into a 2D UV space with aligned textures, enabling effective texture transfer, synthesis, and 3D reconstruction by leveraging well-studied 2D image generative models.

Contribution

It introduces a novel unsupervised UV mapping approach that aligns textures across objects, improving texture transfer and synthesis for 3D shapes.

Findings

Effective texture alignment across objects.

Improved texture transfer and synthesis results.

Versatile applications including 3D reconstruction.

Abstract

In this paper, we address the problem of texture representation for 3D shapes for the challenging and underexplored tasks of texture transfer and synthesis. Previous works either apply spherical texture maps which may lead to large distortions, or use continuous texture fields that yield smooth outputs lacking details. We argue that the traditional way of representing textures with images and linking them to a 3D mesh via UV mapping is more desirable, since synthesizing 2D images is a well-studied problem. We propose AUV-Net which learns to embed 3D surfaces into a 2D aligned UV space, by mapping the corresponding semantic parts of different 3D shapes to the same location in the UV space. As a result, textures are aligned across objects, and can thus be easily synthesized by generative models of images. Texture alignment is learned in an unsupervised manner by a simple yet effective texture alignment module, taking inspiration from traditional works on linear subspace learning. The learned UV mapping and aligned texture representations enable a variety of applications including texture transfer, texture synthesis, and textured single view 3D reconstruction. We conduct experiments on multiple datasets to demonstrate the effectiveness of our method. Project page: https://nv-tlabs.github.io/AUV-NET.