StyleGAN-induced data-driven regularization for inverse problems

TL;DR

This paper introduces L-BRGM, a Bayesian image reconstruction method that leverages a pre-trained StyleGAN2 to improve inverse problem solutions like inpainting and super-resolution.

Contribution

It develops a novel joint optimization framework that enhances StyleGAN2's expressive power by allowing layer-specific style-codes for better image reconstruction.

Findings

Competitive with state-of-the-art GAN-based methods

Superior in some cases for inpainting and super-resolution

Demonstrates the effectiveness of layer-specific style-codes

Abstract

Recent advances in generative adversarial networks (GANs) have opened up the possibility of generating high-resolution photo-realistic images that were impossible to produce previously. The ability of GANs to sample from high-dimensional distributions has naturally motivated researchers to leverage their power for modeling the image prior in inverse problems. We extend this line of research by developing a Bayesian image reconstruction framework that utilizes the full potential of a pre-trained StyleGAN2 generator, which is the currently dominant GAN architecture, for constructing the prior distribution on the underlying image. Our proposed approach, which we refer to as learned Bayesian reconstruction with generative models (L-BRGM), entails joint optimization over the style-code and the input latent code, and enhances the expressive power of a pre-trained StyleGAN2 generator by allowing the style-codes to be different for different generator layers. Considering the inverse problems of image inpainting and super-resolution, we demonstrate that the proposed approach is competitive with, and sometimes superior to, state-of-the-art GAN-based image reconstruction methods.