Learning a Mixture of Deep Networks for Single Image Super-Resolution

TL;DR

This paper introduces a unified framework that learns a mixture of specialized deep networks for single image super-resolution, achieving state-of-the-art results by adaptively aggregating multiple HR estimates.

Contribution

It proposes a novel mixture of deep networks approach for super-resolution, enabling adaptive aggregation of multiple local pattern-specific modules within a joint optimization framework.

Findings

Achieves state-of-the-art super-resolution results.

Flexible network design allows adaptation to different image patterns.

Joint training improves overall super-resolution performance.

Abstract

Single image super-resolution (SR) is an ill-posed problem which aims to recover high-resolution (HR) images from their low-resolution (LR) observations. The crux of this problem lies in learning the complex mapping between low-resolution patches and the corresponding high-resolution patches. Prior arts have used either a mixture of simple regression models or a single non-linear neural network for this propose. This paper proposes the method of learning a mixture of SR inference modules in a unified framework to tackle this problem. Specifically, a number of SR inference modules specialized in different image local patterns are first independently applied on the LR image to obtain various HR estimates, and the resultant HR estimates are adaptively aggregated to form the final HR image. By selecting neural networks as the SR inference module, the whole procedure can be incorporated into a unified network and be optimized jointly. Extensive experiments are conducted to investigate the relation between restoration performance and different network architectures. Compared with other current image SR approaches, our proposed method achieves state-of-the-arts restoration results on a wide range of images consistently while allowing more flexible design choices. The source codes are available in http://www.ifp.illinois.edu/~dingliu2/accv2016.