Scale-wise Convolution for Image Restoration

TL;DR

This paper introduces scale-wise convolution, a novel neural network component that enhances image restoration tasks by effectively modeling scale-invariance, leading to improved accuracy and efficiency across multiple restoration applications.

Contribution

The paper proposes scale-wise convolution, a new method for integrating scale-invariance into neural networks for image restoration, demonstrating significant performance gains.

Findings

Outperforms existing multi-scale neural networks in restoration accuracy

Achieves better parameter efficiency in image restoration tasks

Improves performance in super-resolution, denoising, and compression artifacts removal

Abstract

While scale-invariant modeling has substantially boosted the performance of visual recognition tasks, it remains largely under-explored in deep networks based image restoration. Naively applying those scale-invariant techniques (e.g. multi-scale testing, random-scale data augmentation) to image restoration tasks usually leads to inferior performance. In this paper, we show that properly modeling scale-invariance into neural networks can bring significant benefits to image restoration performance. Inspired from spatial-wise convolution for shift-invariance, "scale-wise convolution" is proposed to convolve across multiple scales for scale-invariance. In our scale-wise convolutional network (SCN), we first map the input image to the feature space and then build a feature pyramid representation via bi-linear down-scaling progressively. The feature pyramid is then passed to a residual network with scale-wise convolutions. The proposed scale-wise convolution learns to dynamically activate and aggregate features from different input scales in each residual building block, in order to exploit contextual information on multiple scales. In experiments, we compare the restoration accuracy and parameter efficiency among our model and many different variants of multi-scale neural networks. The proposed network with scale-wise convolution achieves superior performance in multiple image restoration tasks including image super-resolution, image denoising and image compression artifacts removal. Code and models are available at: https://github.com/ychfan/scn_sr