Super-Resolution with Deep Convolutional Sufficient Statistics

TL;DR

This paper introduces a deep convolutional sufficient statistics model using Gibbs distribution for super-resolution, capturing complex distributions and multi-modality better than point estimates, with potential applications in other ill-posed inverse problems.

Contribution

It proposes a novel deep convolutional sufficient statistics approach with a Gibbs distribution for high-dimensional structured prediction, improving modeling of complex, multi-modal distributions.

Findings

Effective in image super-resolution tasks

Reduces uncertainty in target signals

Potential applicability to audio bandwidth extension

Abstract

Inverse problems in image and audio, and super-resolution in particular, can be seen as high-dimensional structured prediction problems, where the goal is to characterize the conditional distribution of a high-resolution output given its low-resolution corrupted observation. When the scaling ratio is small, point estimates achieve impressive performance, but soon they suffer from the regression-to-the-mean problem, result of their inability to capture the multi-modality of this conditional distribution. Modeling high-dimensional image and audio distributions is a hard task, requiring both the ability to model complex geometrical structures and textured regions. In this paper, we propose to use as conditional model a Gibbs distribution, where its sufficient statistics are given by deep convolutional neural networks. The features computed by the network are stable to local deformation, and have reduced variance when the input is a stationary texture. These properties imply that the resulting sufficient statistics minimize the uncertainty of the target signals given the degraded observations, while being highly informative. The filters of the CNN are initialized by multiscale complex wavelets, and then we propose an algorithm to fine-tune them by estimating the gradient of the conditional log-likelihood, which bears some similarities with Generative Adversarial Networks. We evaluate experimentally the proposed approach in the image super-resolution task, but the approach is general and could be used in other challenging ill-posed problems such as audio bandwidth extension.