Fast and High-Quality Image Denoising via Malleable Convolutions

TL;DR

This paper introduces Malleable Convolution (MalleConv), a novel spatially-varying convolution method that balances efficiency and adaptability, enabling faster high-quality image denoising with minimal computational overhead.

Contribution

The paper proposes MalleConv, a new efficient spatially-varying convolution technique, and demonstrates its effectiveness in building a fast, high-quality denoising network called MalleNet.

Findings

MalleNet is 8.9x faster than top denoising algorithms with similar quality.

MalleConv significantly reduces computational cost when added to standard backbones.

MalleConv enhances receptive field without high computational overhead.

Abstract

Most image denoising networks apply a single set of static convolutional kernels across the entire input image. This is sub-optimal for natural images, as they often consist of heterogeneous visual patterns. Dynamic convolution tries to address this issue by using per-pixel convolution kernels, but this greatly increases computational cost. In this work, we present Malleable Convolution (MalleConv), which performs spatial-varying processing with minimal computational overhead. MalleConv uses a smaller set of spatially-varying convolution kernels, a compromise between static and per-pixel convolution kernels. These spatially-varying kernels are produced by an efficient predictor network running on a downsampled input, making them much more efficient to compute than per-pixel kernels produced by a full-resolution image, and also enlarging the network's receptive field compared with static kernels. These kernels are then jointly upsampled and applied to a full-resolution feature map through an efficient on-the-fly slicing operator with minimum memory overhead. To demonstrate the effectiveness of MalleConv, we use it to build an efficient denoising network we call MalleNet. MalleNet achieves high-quality results without very deep architectures, making it 8.9x faster than the best performing denoising algorithms while achieving similar visual quality. We also show that a single MalleConv layer added to a standard convolution-based backbone can significantly reduce the computational cost or boost image quality at a similar cost. More information is on our project page: \url{https://yifanjiang.net/MalleConv.html}