StyleGAN knows Normal, Depth, Albedo, and More

TL;DR

This paper shows that StyleGAN can generate intrinsic images like depth, normal, and albedo by applying fixed offsets in the latent space, offering a new way to produce and understand scene properties.

Contribution

It introduces a simple method to induce StyleGAN to produce intrinsic images through fixed latent space offsets, revealing the model's inherent understanding of scene properties.

Findings

StyleGAN can produce various intrinsic images with fixed latent offsets.

Intrinsic images from StyleGAN are robust to relighting effects.

StyleGAN's intrinsic images compare well with state-of-the-art regression methods.

Abstract

Intrinsic images, in the original sense, are image-like maps of scene properties like depth, normal, albedo or shading. This paper demonstrates that StyleGAN can easily be induced to produce intrinsic images. The procedure is straightforward. We show that, if StyleGAN produces $G({w})$ from latents ${w}$, then for each type of intrinsic image, there is a fixed offset ${d}_c$ so that $G({w}+{d}_c)$ is that type of intrinsic image for $G({w})$. Here ${d}_c$ is {\em independent of ${w}$}. The StyleGAN we used was pretrained by others, so this property is not some accident of our training regime. We show that there are image transformations StyleGAN will {\em not} produce in this fashion, so StyleGAN is not a generic image regression engine. It is conceptually exciting that an image generator should ``know'' and represent intrinsic images. There may also be practical advantages to using a generative model to produce intrinsic images. The intrinsic images obtained from StyleGAN compare well both qualitatively and quantitatively with those obtained by using SOTA image regression techniques; but StyleGAN's intrinsic images are robust to relighting effects, unlike SOTA methods.