SIRA: Relightable Avatars from a Single Image

TL;DR

SIRA is a novel method that reconstructs high-fidelity 3D human head avatars from a single image, disentangling geometry, materials, and lighting for realistic relighting and editing.

Contribution

It introduces a data-driven neural field approach for single-image 3D head reconstruction with disentangled appearance and lighting, surpassing previous methods in fidelity and versatility.

Findings

Achieves state-of-the-art 3D head reconstruction from one image.

Successfully disentangles global illumination, diffuse, and specular components.

Enables physically-based appearance editing and relighting.

Abstract

Recovering the geometry of a human head from a single image, while factorizing the materials and illumination is a severely ill-posed problem that requires prior information to be solved. Methods based on 3D Morphable Models (3DMM), and their combination with differentiable renderers, have shown promising results. However, the expressiveness of 3DMMs is limited, and they typically yield over-smoothed and identity-agnostic 3D shapes limited to the face region. Highly accurate full head reconstructions have recently been obtained with neural fields that parameterize the geometry using multilayer perceptrons. The versatility of these representations has also proved effective for disentangling geometry, materials and lighting. However, these methods require several tens of input images. In this paper, we introduce SIRA, a method which, from a single image, reconstructs human head avatars with high fidelity geometry and factorized lights and surface materials. Our key ingredients are two data-driven statistical models based on neural fields that resolve the ambiguities of single-view 3D surface reconstruction and appearance factorization. Experiments show that SIRA obtains state of the art results in 3D head reconstruction while at the same time it successfully disentangles the global illumination, and the diffuse and specular albedos. Furthermore, our reconstructions are amenable to physically-based appearance editing and head model relighting.