TexMesh: Reconstructing Detailed Human Texture and Geometry from RGB-D Video

TL;DR

TexMesh is a new method for reconstructing detailed 3D human meshes with high-resolution textures from RGB-D video, enabling realistic free-viewpoint rendering.

Contribution

It introduces a self-supervised approach that estimates detailed geometry and textures using photometric constraints and incident illumination, adapting models to real-world data.

Findings

Outperforms state-of-the-art methods quantitatively.

Produces high-quality, detailed human meshes with textures.

Operates at interactive framerate after training.

Abstract

We present TexMesh, a novel approach to reconstruct detailed human meshes with high-resolution full-body texture from RGB-D video. TexMesh enables high quality free-viewpoint rendering of humans. Given the RGB frames, the captured environment map, and the coarse per-frame human mesh from RGB-D tracking, our method reconstructs spatiotemporally consistent and detailed per-frame meshes along with a high-resolution albedo texture. By using the incident illumination we are able to accurately estimate local surface geometry and albedo, which allows us to further use photometric constraints to adapt a synthetically trained model to real-world sequences in a self-supervised manner for detailed surface geometry and high-resolution texture estimation. In practice, we train our models on a short example sequence for self-adaptation and the model runs at interactive framerate afterwards. We validate TexMesh on synthetic and real-world data, and show it outperforms the state of art quantitatively and qualitatively.