On the relation between statistical learning and perceptual distances

TL;DR

This paper investigates how perceptual distances relate to data distribution and their impact on machine learning, revealing correlations with image probability and conditions where perceptual distances improve performance.

Contribution

It uncovers the relationship between perceptual distances, data probability, and autoencoder-induced metrics, and explains when perceptual distances enhance image processing tasks.

Findings

Perceptual sensitivity correlates with image probability in local neighborhoods.

Distances from autoencoders relate to training data distribution.

Perceptual distances offer limited gains unless data is scarce, due to double-counting effects.

Abstract

It has been demonstrated many times that the behavior of the human visual system is connected to the statistics of natural images. Since machine learning relies on the statistics of training data as well, the above connection has interesting implications when using perceptual distances (which mimic the behavior of the human visual system) as a loss function. In this paper, we aim to unravel the non-trivial relationships between the probability distribution of the data, perceptual distances, and unsupervised machine learning. To this end, we show that perceptual sensitivity is correlated with the probability of an image in its close neighborhood. We also explore the relation between distances induced by autoencoders and the probability distribution of the training data, as well as how these induced distances are correlated with human perception. Finally, we find perceptual distances do not always lead to noticeable gains in performance over Euclidean distance in common image processing tasks, except when data is scarce and the perceptual distance provides regularization. We propose this may be due to a \emph{double-counting} effect of the image statistics, once in the perceptual distance and once in the training procedure.