Generic Perceptual Loss for Modeling Structured Output Dependencies

TL;DR

This paper reveals that the structure of a deep CNN, even without training, can effectively model structured output dependencies in image synthesis tasks, removing the need for pre-trained weights.

Contribution

It introduces a novel perceptual loss based on randomly-weighted CNNs, broadening the applicability of perceptual losses to various structured output learning tasks.

Findings

Randomly-weighted CNNs can model output dependencies effectively.

Extended perceptual loss improves dense prediction task results.

No pre-training or specific network structure needed.

Abstract

The perceptual loss has been widely used as an effective loss term in image synthesis tasks including image super-resolution, and style transfer. It was believed that the success lies in the high-level perceptual feature representations extracted from CNNs pretrained with a large set of images. Here we reveal that, what matters is the network structure instead of the trained weights. Without any learning, the structure of a deep network is sufficient to capture the dependencies between multiple levels of variable statistics using multiple layers of CNNs. This insight removes the requirements of pre-training and a particular network structure (commonly, VGG) that are previously assumed for the perceptual loss, thus enabling a significantly wider range of applications. To this end, we demonstrate that a randomly-weighted deep CNN can be used to model the structured dependencies of outputs. On a few dense per-pixel prediction tasks such as semantic segmentation, depth estimation and instance segmentation, we show improved results of using the extended randomized perceptual loss, compared to the baselines using pixel-wise loss alone. We hope that this simple, extended perceptual loss may serve as a generic structured-output loss that is applicable to most structured output learning tasks.