Exploring the Low-Pass Filtering Behavior in Image Super-Resolution

TL;DR

This paper investigates the low-pass filtering behavior of deep neural networks in image super-resolution, revealing that they act as low-pass filters with high-frequency injection, and introduces new analysis tools and metrics for understanding this behavior.

Contribution

It proposes Hybrid Response Analysis (HyRA) to decompose neural networks into linear and non-linear parts, and introduces FSDS to quantify high-frequency information injection.

Findings

Neural networks exhibit sinc phenomenon with impulse inputs.

Linear components act as low-pass filters in ISR.

High-frequency information is injected by non-linear components.

Abstract

Deep neural networks for image super-resolution (ISR) have shown significant advantages over traditional approaches like the interpolation. However, they are often criticized as 'black boxes' compared to traditional approaches with solid mathematical foundations. In this paper, we attempt to interpret the behavior of deep neural networks in ISR using theories from the field of signal processing. First, we report an intriguing phenomenon, referred to as `the sinc phenomenon.' It occurs when an impulse input is fed to a neural network. Then, building on this observation, we propose a method named Hybrid Response Analysis (HyRA) to analyze the behavior of neural networks in ISR tasks. Specifically, HyRA decomposes a neural network into a parallel connection of a linear system and a non-linear system and demonstrates that the linear system functions as a low-pass filter while the non-linear system injects high-frequency information. Finally, to quantify the injected high-frequency information, we introduce a metric for image-to-image tasks called Frequency Spectrum Distribution Similarity (FSDS). FSDS reflects the distribution similarity of different frequency components and can capture nuances that traditional metrics may overlook. Code, videos and raw experimental results for this paper can be found in: https://github.com/RisingEntropy/LPFInISR.