FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer

TL;DR

FaceRefiner introduces a novel style transfer approach using differentiable rendering to enhance facial textures, preserving details and identity from in-the-wild images for more realistic 3D face reconstructions.

Contribution

It proposes a multi-level style transfer method with differentiable rendering that improves facial texture quality and identity preservation over existing techniques.

Findings

Enhanced texture quality demonstrated on multiple datasets.

Improved identity preservation compared to state-of-the-art methods.

Effective transfer of low-level details in facial textures.

Abstract

Recent facial texture generation methods prefer to use deep networks to synthesize image content and then fill in the UV map, thus generating a compelling full texture from a single image. Nevertheless, the synthesized texture UV map usually comes from a space constructed by the training data or the 2D face generator, which limits the methods' generalization ability for in-the-wild input images. Consequently, their facial details, structures and identity may not be consistent with the input. In this paper, we address this issue by proposing a style transfer-based facial texture refinement method named FaceRefiner. FaceRefiner treats the 3D sampled texture as style and the output of a texture generation method as content. The photo-realistic style is then expected to be transferred from the style image to the content image. Different from current style transfer methods that only transfer high and middle level information to the result, our style transfer method integrates differentiable rendering to also transfer low level (or pixel level) information in the visible face regions. The main benefit of such multi-level information transfer is that, the details, structures and semantics in the input can thus be well preserved. The extensive experiments on Multi-PIE, CelebA and FFHQ datasets demonstrate that our refinement method can improve the texture quality and the face identity preserving ability, compared with state-of-the-arts.