StyLitGAN: Prompting StyleGAN to Produce New Illumination Conditions

TL;DR

StyLitGAN introduces a new technique for relighting images by manipulating StyleGAN's latent space, enabling realistic lighting effects without labeled data or paired images.

Contribution

It presents a novel latent space search method for relighting and resurfacing images using StyleGAN without requiring paired or CGI data.

Findings

Generates realistic lighting effects including shadows and reflections

Achieves diverse relighting by manipulating latent directions

Qualitative evaluation confirms effectiveness

Abstract

We propose a novel method, StyLitGAN, for relighting and resurfacing generated images in the absence of labeled data. Our approach generates images with realistic lighting effects, including cast shadows, soft shadows, inter-reflections, and glossy effects, without the need for paired or CGI data. StyLitGAN uses an intrinsic image method to decompose an image, followed by a search of the latent space of a pre-trained StyleGAN to identify a set of directions. By prompting the model to fix one component (e.g., albedo) and vary another (e.g., shading), we generate relighted images by adding the identified directions to the latent style codes. Quantitative metrics of change in albedo and lighting diversity allow us to choose effective directions using a forward selection process. Qualitative evaluation confirms the effectiveness of our method.