SSL: A Self-similarity Loss for Improving Generative Image Super-resolution

TL;DR

This paper introduces a self-similarity loss (SSL) that leverages natural image self-similarities to improve the realism and detail of super-resolved images generated by GAN and diffusion models, reducing artifacts.

Contribution

The paper proposes a novel self-similarity loss that enhances super-resolution models by enforcing structural consistency, applicable as a plug-and-play module for various generative models.

Findings

SSL improves perceptual quality of super-resolved images.

SSL reduces visual artifacts and false structures.

State-of-the-art models benefit from significant performance gains.

Abstract

Generative adversarial networks (GAN) and generative diffusion models (DM) have been widely used in real-world image super-resolution (Real-ISR) to enhance the image perceptual quality. However, these generative models are prone to generating visual artifacts and false image structures, resulting in unnatural Real-ISR results. Based on the fact that natural images exhibit high self-similarities, i.e., a local patch can have many similar patches to it in the whole image, in this work we propose a simple yet effective self-similarity loss (SSL) to improve the performance of generative Real-ISR models, enhancing the hallucination of structural and textural details while reducing the unpleasant visual artifacts. Specifically, we compute a self-similarity graph (SSG) of the ground-truth image, and enforce the SSG of Real-ISR output to be close to it. To reduce the training cost and focus on edge areas, we generate an edge mask from the ground-truth image, and compute the SSG only on the masked pixels. The proposed SSL serves as a general plug-and-play penalty, which could be easily applied to the off-the-shelf Real-ISR models. Our experiments demonstrate that, by coupling with SSL, the performance of many state-of-the-art Real-ISR models, including those GAN and DM based ones, can be largely improved, reproducing more perceptually realistic image details and eliminating many false reconstructions and visual artifacts. Codes and supplementary material can be found at https://github.com/ChrisDud0257/SSL