Maintaining Natural Image Statistics with the Contextual Loss

TL;DR

This paper introduces a new method for training CNNs to preserve natural image statistics, improving realism in image restoration and generation without solely relying on adversarial training.

Contribution

A complementary approach to GANs that trains CNNs to maintain natural feature distributions, reducing training data needs and achieving state-of-the-art results.

Findings

Reduces training data requirements by orders of magnitude.

Achieves state-of-the-art results in super-resolution.

Effective in high-resolution surface normal estimation.

Abstract

Maintaining natural image statistics is a crucial factor in restoration and generation of realistic looking images. When training CNNs, photorealism is usually attempted by adversarial training (GAN), that pushes the output images to lie on the manifold of natural images. GANs are very powerful, but not perfect. They are hard to train and the results still often suffer from artifacts. In this paper we propose a complementary approach, that could be applied with or without GAN, whose goal is to train a feed-forward CNN to maintain natural internal statistics. We look explicitly at the distribution of features in an image and train the network to generate images with natural feature distributions. Our approach reduces by orders of magnitude the number of images required for training and achieves state-of-the-art results on both single-image super-resolution, and high-resolution surface normal estimation.