JIFF: Jointly-aligned Implicit Face Function for High Quality Single View Clothed Human Reconstruction

TL;DR

This paper introduces JIFF, a novel method that combines implicit functions and model-based approaches to significantly improve the quality of face details in single-view 3D human reconstructions, especially for applications like telepresence.

Contribution

The paper proposes a joint implicit face function using a 3D morphable model and space-aligned features, enhancing face detail accuracy in 3D human reconstructions.

Findings

Superior face detail reconstruction compared to existing methods

Effective integration of 3D morphable model with implicit functions

High-quality texture prediction with coarse-to-fine architecture

Abstract

This paper addresses the problem of single view 3D human reconstruction. Recent implicit function based methods have shown impressive results, but they fail to recover fine face details in their reconstructions. This largely degrades user experience in applications like 3D telepresence. In this paper, we focus on improving the quality of face in the reconstruction and propose a novel Jointly-aligned Implicit Face Function (JIFF) that combines the merits of the implicit function based approach and model based approach. We employ a 3D morphable face model as our shape prior and compute space-aligned 3D features that capture detailed face geometry information. Such space-aligned 3D features are combined with pixel-aligned 2D features to jointly predict an implicit face function for high quality face reconstruction. We further extend our pipeline and introduce a coarse-to-fine architecture to predict high quality texture for our detailed face model. Extensive evaluations have been carried out on public datasets and our proposed JIFF has demonstrates superior performance (both quantitatively and qualitatively) over existing state-of-the-arts.