High-Resolution GAN Inversion for Degraded Images in Large Diverse Datasets

TL;DR

This paper introduces a novel GAN inversion framework using StyleGAN-XL and clustering-based regularization to effectively restore high-quality images from various degraded natural images across multiple tasks.

Contribution

The paper proposes a new CRI scheme that improves GAN inversion for degraded images by clustering latent spaces and regularizing the inversion process, applied to multiple restoration tasks.

Findings

CRI outperforms existing methods in image restoration quality

The approach is robust across different data types and GAN models

First to utilize StyleGAN-XL for high-quality natural image restoration from degraded inputs

Abstract

The last decades are marked by massive and diverse image data, which shows increasingly high resolution and quality. However, some images we obtained may be corrupted, affecting the perception and the application of downstream tasks. A generic method for generating a high-quality image from the degraded one is in demand. In this paper, we present a novel GAN inversion framework that utilizes the powerful generative ability of StyleGAN-XL for this problem. To ease the inversion challenge with StyleGAN-XL, Clustering \& Regularize Inversion (CRI) is proposed. Specifically, the latent space is firstly divided into finer-grained sub-spaces by clustering. Instead of initializing the inversion with the average latent vector, we approximate a centroid latent vector from the clusters, which generates an image close to the input image. Then, an offset with a regularization term is introduced to keep the inverted latent vector within a certain range. We validate our CRI scheme on multiple restoration tasks (i.e., inpainting, colorization, and super-resolution) of complex natural images, and show preferable quantitative and qualitative results. We further demonstrate our technique is robust in terms of data and different GAN models. To our best knowledge, we are the first to adopt StyleGAN-XL for generating high-quality natural images from diverse degraded inputs. Code is available at https://github.com/Booooooooooo/CRI.