LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements

TL;DR

LEGAN is a novel face image synthesis framework that disentangles lighting and expression, leveraging perceptual judgments to generate high-quality, natural-looking images without paired data, improving downstream recognition tasks.

Contribution

This work introduces LEGAN, a framework that jointly manipulates lighting and expressions using perceptual quality guidance, without needing paired training data.

Findings

LEGAN outperforms StarGAN and StarGAN-v2 in image quality metrics.

Perceptual quality estimation correlates with visual fidelity.

Using LEGAN as data augmentation improves expression recognition and face verification.

Abstract

Building facial analysis systems that generalize to extreme variations in lighting and facial expressions is a challenging problem that can potentially be alleviated using natural-looking synthetic data. Towards that, we propose LEGAN, a novel synthesis framework that leverages perceptual quality judgments for jointly manipulating lighting and expressions in face images, without requiring paired training data. LEGAN disentangles the lighting and expression subspaces and performs transformations in the feature space before upscaling to the desired output image. The fidelity of the synthetic image is further refined by integrating a perceptual quality estimation model, trained with face images rendered using multiple synthesis methods and their crowd-sourced naturalness ratings, into the LEGAN framework as an auxiliary discriminator. Using objective metrics like FID and LPIPS, LEGAN is shown to generate higher quality face images when compared with popular GAN models like StarGAN and StarGAN-v2 for lighting and expression synthesis. We also conduct a perceptual study using images synthesized by LEGAN and other GAN models and show the correlation between our quality estimation and visual fidelity. Finally, we demonstrate the effectiveness of LEGAN as training data augmenter for expression recognition and face verification tasks.