Image Completion with Heterogeneously Filtered Spectral Hints

TL;DR

This paper introduces SH-GAN, a spectral hint-based image completion model that leverages spectral processing strategies to improve texture clarity and structural accuracy, achieving state-of-the-art results on benchmark datasets.

Contribution

The paper proposes a novel spectral processing module and strategies within a StyleGAN framework for improved large-scale image completion.

Findings

Achieves new state-of-the-art FID scores on FFHQ and Places2 datasets.

Effectively resolves pattern unawareness, blurry textures, and structure distortion.

Demonstrates the effectiveness of spectral processing strategies through ablation studies.

Abstract

Image completion with large-scale free-form missing regions is one of the most challenging tasks for the computer vision community. While researchers pursue better solutions, drawbacks such as pattern unawareness, blurry textures, and structure distortion remain noticeable, and thus leave space for improvement. To overcome these challenges, we propose a new StyleGAN-based image completion network, Spectral Hint GAN (SH-GAN), inside which a carefully designed spectral processing module, Spectral Hint Unit, is introduced. We also propose two novel 2D spectral processing strategies, Heterogeneous Filtering and Gaussian Split that well-fit modern deep learning models and may further be extended to other tasks. From our inclusive experiments, we demonstrate that our model can reach FID scores of 3.4134 and 7.0277 on the benchmark datasets FFHQ and Places2, and therefore outperforms prior works and reaches a new state-of-the-art. We also prove the effectiveness of our design via ablation studies, from which one may notice that the aforementioned challenges, i.e. pattern unawareness, blurry textures, and structure distortion, can be noticeably resolved. Our code will be open-sourced at: https://github.com/SHI-Labs/SH-GAN.