VoLux-GAN: A Generative Model for 3D Face Synthesis with HDRI Relighting

TL;DR

VoLux-GAN is a novel 3D face synthesis framework that enables realistic relighting by integrating volumetric HDRI techniques and supervised image decomposition, advancing photorealistic 3D generative modeling.

Contribution

The paper introduces a volumetric HDRI relighting method and a supervised image decomposition approach, enhancing 3D face synthesis with convincing relighting capabilities.

Findings

Effective accumulation of lighting contributions along 3D rays.

Supervised decomposition improves realism and consistency.

Outperforms existing frameworks in photorealistic relighting.

Abstract

We propose VoLux-GAN, a generative framework to synthesize 3D-aware faces with convincing relighting. Our main contribution is a volumetric HDRI relighting method that can efficiently accumulate albedo, diffuse and specular lighting contributions along each 3D ray for any desired HDR environmental map. Additionally, we show the importance of supervising the image decomposition process using multiple discriminators. In particular, we propose a data augmentation technique that leverages recent advances in single image portrait relighting to enforce consistent geometry, albedo, diffuse and specular components. Multiple experiments and comparisons with other generative frameworks show how our model is a step forward towards photorealistic relightable 3D generative models.